You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Kolhoff, Jacqueline - ENCOWAY" <Ko...@encoway.de> on 2010/07/01 11:19:29 UTC

Lucene and Chinese language

Hi!

We are using lucene in our project to search through information objects which works fine. For indexing we use the StandardAnalyzer.
Now, we have to support the Chinese language. I found out that the Chinese words and letters are correctly saved in the index but the query to search for them does not work. Example: in English language the query is “text” which we parse to “*text*”. If we search for Chinese words / phrases like “佛山东方书城”the query is “*佛山东方书城*“ but there are no search results. If the query places blanks between the single letters / symbols like this “*佛 山 东 方 书 城*“ we are getting results. Does the StandardAnalyzer interpret each Chinese letter as one word? What are best practices for this case? Shall we use another analyzer (Chinese analyzer)? Or is it better to replace the query parser in this case?

Regards,
Jacqueline.

Re: Lucene and Chinese language

Posted by lwl <lw...@gmail.com>.
yes, the StandardAnalyzer interpret each Chinese letter as one word.
Better analyzers for chinese are here:
http://hi.baidu.com/lewutian/blog/item/ca61060a06914b1394ca6b25.html

在 2010年7月1日 下午5:19,Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>写道:

>
> Hi!
>
> We are using lucene in our project to search through information objects
> which works fine. For indexing we use the StandardAnalyzer.
> Now, we have to support the Chinese language. I found out that the Chinese
> words and letters are correctly saved in the index but the query to search
> for them does not work. Example: in English language the query is “text”
> which we parse to “*text*”. If we search for Chinese words / phrases like
> “佛山东方书城”the query is “*佛山东方书城*“ but there are no search results. If the
> query places blanks between the single letters / symbols like this “*佛 山 东 方
> 书 城*“ we are getting results. Does the StandardAnalyzer interpret each
> Chinese letter as one word? What are best practices for this case? Shall we
> use another analyzer (Chinese analyzer)? Or is it better to replace the
> query parser in this case?
>
> Regards,
> Jacqueline.
>

Re: Lucene and Chinese language

Posted by Danil ŢORIN <to...@gmail.com>.
Try to use CJK analyzer for both indexing and searching chinese language.
Then you won't need "text"->"*text*" transformation.

There might be some false positives in the results though.
You can also may want to try smartcn analyzer which is dictionary based, but
I have no expertise to evaluate the results (we still use CJK for asian
languages, as there are no complains so far)


2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>

>
> Hi!
>
> We are using lucene in our project to search through information objects
> which works fine. For indexing we use the StandardAnalyzer.
> Now, we have to support the Chinese language. I found out that the Chinese
> words and letters are correctly saved in the index but the query to search
> for them does not work. Example: in English language the query is “text”
> which we parse to “*text*”. If we search for Chinese words / phrases like
> “佛山东方书城”the query is “*佛山东方书城*“ but there are no search results. If the
> query places blanks between the single letters / symbols like this “*佛 山 东 方
> 书 城*“ we are getting results. Does the StandardAnalyzer interpret each
> Chinese letter as one word? What are best practices for this case? Shall we
> use another analyzer (Chinese analyzer)? Or is it better to replace the
> query parser in this case?
>
> Regards,
> Jacqueline.
>

Re: Lucene and Chinese language

Posted by kafka0102 <ka...@163.com>.
you can choose ik : http://code.google.com/p/ik-analyzer/ for this 
problem. It generates a better token result than standard and CJK. And, 
you can use its IKQueryParser instead of queryparser. It generates "and 
queries" instead of " phrase queries".



On 2010年07月01日 20:47, Robert Muir wrote:
> its really just a bad situation for chinese :(
>
> with queryparser, you either get no phrase query support (by using the
> hack), or all queries automatically become phrase queries.
>
> if you want to do the equivalent of *在电力虎*, then you need to use queryparser
> without the hack, just do a query of "在电力虎" (with quotes).
> if you want to do the equivalent of 在 电 力 虎, then you need to use
> queryparser with the hack, just do a query of 在电力虎 (without quotes).
>
>
> 2010/7/1 Kolhoff, Jacqueline - ENCOWAY<Ko...@encoway.de>
>
>    
>> Ok, understand!
>>
>> So it is better to use another analyzer in the chinese case at index-time
>> or do you suggest to use another "QueryParser" at query-time?
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Robert Muir [mailto:rcmuir@gmail.com]
>> Gesendet: Donnerstag, 1. Juli 2010 14:35
>> An: java-user@lucene.apache.org
>> Betreff: Re: Lucene and Chinese language
>>
>> 2010/7/1 Kolhoff, Jacqueline - ENCOWAY<Ko...@encoway.de>
>>
>>      
>>> As you can see, the query parser automatically added double quotes and
>>> blanks. But this does not work for our English or German queries.
>>>
>>> If I use the PositionHackAnalyzerWrapper and the case with * I got no
>>> results, query is:
>>> +anotherfieldname:description +myfieldname:*在电力虎*
>>>
>>> If I remove the * the query is:
>>> + anotherfieldname: description
>>> +(myfieldname:在myfieldname:电myfieldname:力myfieldname:虎)
>>>
>>> and I got results but not for German or English queries.
>>>
>>>        
>> Weird?
>>      
>>>
>>>        
>> its working correctly, your chinese wildcard query doesnt make sense, as
>> you
>> havent indexed the text in a way to do queries like that (you have indexed
>> individual chars).
>> in practice this is where you would do a chinese phrase query of "在电力虎"
>> (with quotes) instead of *... but if you use the positionfilterhack, you
>> cant do phrase queries.
>>
>> --
>> Robert Muir
>> rcmuir@gmail.com
>>
>>      
>
>
>    



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Lucene and Chinese language

Posted by Robert Muir <rc...@gmail.com>.
its really just a bad situation for chinese :(

with queryparser, you either get no phrase query support (by using the
hack), or all queries automatically become phrase queries.

if you want to do the equivalent of *在电力虎*, then you need to use queryparser
without the hack, just do a query of "在电力虎" (with quotes).
if you want to do the equivalent of 在 电 力 虎, then you need to use
queryparser with the hack, just do a query of 在电力虎 (without quotes).


2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>

> Ok, understand!
>
> So it is better to use another analyzer in the chinese case at index-time
> or do you suggest to use another "QueryParser" at query-time?
>
> -----Ursprüngliche Nachricht-----
> Von: Robert Muir [mailto:rcmuir@gmail.com]
> Gesendet: Donnerstag, 1. Juli 2010 14:35
> An: java-user@lucene.apache.org
> Betreff: Re: Lucene and Chinese language
>
> 2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>
>
> > As you can see, the query parser automatically added double quotes and
> > blanks. But this does not work for our English or German queries.
> >
> > If I use the PositionHackAnalyzerWrapper and the case with * I got no
> > results, query is:
> > +anotherfieldname:description +myfieldname:*在电力虎*
> >
> > If I remove the * the query is:
> > + anotherfieldname: description
> > +(myfieldname:在myfieldname:电myfieldname:力myfieldname:虎)
> >
> > and I got results but not for German or English queries.
> >
>
> Weird?
> >
> >
> its working correctly, your chinese wildcard query doesnt make sense, as
> you
> havent indexed the text in a way to do queries like that (you have indexed
> individual chars).
> in practice this is where you would do a chinese phrase query of "在电力虎"
> (with quotes) instead of *... but if you use the positionfilterhack, you
> cant do phrase queries.
>
> --
> Robert Muir
> rcmuir@gmail.com
>



-- 
Robert Muir
rcmuir@gmail.com

AW: Lucene and Chinese language

Posted by "Kolhoff, Jacqueline - ENCOWAY" <Ko...@encoway.de>.
Ok, understand!

So it is better to use another analyzer in the chinese case at index-time or do you suggest to use another "QueryParser" at query-time?

-----Ursprüngliche Nachricht-----
Von: Robert Muir [mailto:rcmuir@gmail.com] 
Gesendet: Donnerstag, 1. Juli 2010 14:35
An: java-user@lucene.apache.org
Betreff: Re: Lucene and Chinese language

2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>

> As you can see, the query parser automatically added double quotes and
> blanks. But this does not work for our English or German queries.
>
> If I use the PositionHackAnalyzerWrapper and the case with * I got no
> results, query is:
> +anotherfieldname:description +myfieldname:*在电力虎*
>
> If I remove the * the query is:
> + anotherfieldname: description
> +(myfieldname:在myfieldname:电myfieldname:力myfieldname:虎)
>
> and I got results but not for German or English queries.
>

Weird?
>
>
its working correctly, your chinese wildcard query doesnt make sense, as you
havent indexed the text in a way to do queries like that (you have indexed
individual chars).
in practice this is where you would do a chinese phrase query of "在电力虎"
(with quotes) instead of *... but if you use the positionfilterhack, you
cant do phrase queries.

-- 
Robert Muir
rcmuir@gmail.com

Re: Lucene and Chinese language

Posted by Robert Muir <rc...@gmail.com>.
2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>

> As you can see, the query parser automatically added double quotes and
> blanks. But this does not work for our English or German queries.
>
> If I use the PositionHackAnalyzerWrapper and the case with * I got no
> results, query is:
> +anotherfieldname:description +myfieldname:*在电力虎*
>
> If I remove the * the query is:
> + anotherfieldname: description
> +(myfieldname:在myfieldname:电myfieldname:力myfieldname:虎)
>
> and I got results but not for German or English queries.
>

Weird?
>
>
its working correctly, your chinese wildcard query doesnt make sense, as you
havent indexed the text in a way to do queries like that (you have indexed
individual chars).
in practice this is where you would do a chinese phrase query of "在电力虎"
(with quotes) instead of *... but if you use the positionfilterhack, you
cant do phrase queries.

-- 
Robert Muir
rcmuir@gmail.com

AW: Lucene and Chinese language

Posted by "Kolhoff, Jacqueline - ENCOWAY" <Ko...@encoway.de>.
Hm, that doesn't work for me. 

Maybe we misunderstood:

in my query there aren't any double quotes (just add them to show that this is my query).

I search in a field for a special value (substring) like 在电力虎

My query looks like this:

+anotherfieldname:description +myfieldname:*在电力虎*

We always add the multiple wildcard character (*).
Query String is 在电力虎

With the first way

QueryParser queryParser = new QueryParser(Version.LUCENE_30, "myfieldname"
 , new StandardAnalyzer(Version.LUCENE_30));

we got no results.

If we do not add the * it works with StandardAnalyzer for Chinese, query is:
+anotherfieldname:description +myfieldname:"在 电 力 虎"

As you can see, the query parser automatically added double quotes and blanks. But this does not work for our English or German queries.

If I use the PositionHackAnalyzerWrapper and the case with * I got no results, query is:
+anotherfieldname:description +myfieldname:*在电力虎*

If I remove the * the query is:
+ anotherfieldname: description +(myfieldname:在myfieldname:电myfieldname:力myfieldname:虎)

and I got results but not for German or English queries.

Weird?


-----Ursprüngliche Nachricht-----
Von: Robert Muir [mailto:rcmuir@gmail.com] 
Gesendet: Donnerstag, 1. Juli 2010 13:51
An: java-user@lucene.apache.org
Betreff: Re: Lucene and Chinese language

you can make your own analyzer, or do something like the below at
query-time.

QueryParser queryParser = new QueryParser(Version.LUCENE_30, "myfieldname" ,
new PositionHackAnalyzerWrapper(new StandardAnalyzer(Version.LUCENE_30)));

public class PositionHackAnalyzerWrapper extends Analyzer {
  Analyzer wrapped;

  public PositionHackAnalyzerWrapper(Analyzer wrapped) {
    this.wrapped = wrapped;
  }

  @Override
  public TokenStream tokenStream(String fieldName, Reader reader) {
    TokenStream ts = wrapped.tokenStream(fieldName, reader);
    return new PositionFilter(ts);
  }
}

2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>

> How can I add this PositionFilter? I can't see anything in the API. I use
> lucene version 3.0.1, this is my query parser:
>
> QueryParser queryParser = new QueryParser(Version.LUCENE_30, "myfieldname"
> , new StandardAnalyzer(Version.LUCENE_30));
>
> -----Ursprüngliche Nachricht-----
> Von: Robert Muir [mailto:rcmuir@gmail.com]
> Gesendet: Donnerstag, 1. Juli 2010 12:34
> An: java-user@lucene.apache.org
> Betreff: Re: Lucene and Chinese language
>
> This is a bug in the queryparser. (
> https://issues.apache.org/jira/browse/LUCENE-2458)
>
> the problem has nothing to do with your choice of analyzer, it has to do
> with how the query is formed.
>
> Currently the queryparser uses a convoluted algorithm involving whitespace
> (and not just the double quote operator as you would expect) to form phrase
> queries. So, queries like this with no whitespace form phrase queries
> always.
>
> The only workaround for reasonably good results consists of two steps:
> 1. at query time (only!) add a
> org.apache.lucene.analysis.position.PositionFilter (from contrib/analyzers)
> to your analyzer. don't do this at index-time, just query-time!
> 2. this will make all terms in the query "synonyms" of each other to bypass
> this problem, but will screw up scoring, so you might want to also extend
> QueryParser in a custom way:
>
> @Override
>  protected BooleanQuery newBooleanQuery(boolean disableCoord) {
>   // intentionally ignore disabled
>   // coord() factor from the PositionFilter hack.
>   return new BooleanQuery(false);
>  }
>
> 2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>
>
> >
> > Hi!
> >
> > We are using lucene in our project to search through information objects
> > which works fine. For indexing we use the StandardAnalyzer.
> > Now, we have to support the Chinese language. I found out that the
> Chinese
> > words and letters are correctly saved in the index but the query to
> search
> > for them does not work. Example: in English language the query is “text”
> > which we parse to “*text*”. If we search for Chinese words / phrases like
> > “佛山东方书城”the query is “*佛山东方书城*“ but there are no search results. If the
> > query places blanks between the single letters / symbols like this “*佛 山
> 东 方
> > 书 城*“ we are getting results. Does the StandardAnalyzer interpret each
> > Chinese letter as one word? What are best practices for this case? Shall
> we
> > use another analyzer (Chinese analyzer)? Or is it better to replace the
> > query parser in this case?
> >
> > Regards,
> > Jacqueline.
> >
>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>



-- 
Robert Muir
rcmuir@gmail.com

Re: Lucene and Chinese language

Posted by Robert Muir <rc...@gmail.com>.
you can make your own analyzer, or do something like the below at
query-time.

QueryParser queryParser = new QueryParser(Version.LUCENE_30, "myfieldname" ,
new PositionHackAnalyzerWrapper(new StandardAnalyzer(Version.LUCENE_30)));

public class PositionHackAnalyzerWrapper extends Analyzer {
  Analyzer wrapped;

  public PositionHackAnalyzerWrapper(Analyzer wrapped) {
    this.wrapped = wrapped;
  }

  @Override
  public TokenStream tokenStream(String fieldName, Reader reader) {
    TokenStream ts = wrapped.tokenStream(fieldName, reader);
    return new PositionFilter(ts);
  }
}

2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>

> How can I add this PositionFilter? I can't see anything in the API. I use
> lucene version 3.0.1, this is my query parser:
>
> QueryParser queryParser = new QueryParser(Version.LUCENE_30, "myfieldname"
> , new StandardAnalyzer(Version.LUCENE_30));
>
> -----Ursprüngliche Nachricht-----
> Von: Robert Muir [mailto:rcmuir@gmail.com]
> Gesendet: Donnerstag, 1. Juli 2010 12:34
> An: java-user@lucene.apache.org
> Betreff: Re: Lucene and Chinese language
>
> This is a bug in the queryparser. (
> https://issues.apache.org/jira/browse/LUCENE-2458)
>
> the problem has nothing to do with your choice of analyzer, it has to do
> with how the query is formed.
>
> Currently the queryparser uses a convoluted algorithm involving whitespace
> (and not just the double quote operator as you would expect) to form phrase
> queries. So, queries like this with no whitespace form phrase queries
> always.
>
> The only workaround for reasonably good results consists of two steps:
> 1. at query time (only!) add a
> org.apache.lucene.analysis.position.PositionFilter (from contrib/analyzers)
> to your analyzer. don't do this at index-time, just query-time!
> 2. this will make all terms in the query "synonyms" of each other to bypass
> this problem, but will screw up scoring, so you might want to also extend
> QueryParser in a custom way:
>
> @Override
>  protected BooleanQuery newBooleanQuery(boolean disableCoord) {
>   // intentionally ignore disabled
>   // coord() factor from the PositionFilter hack.
>   return new BooleanQuery(false);
>  }
>
> 2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>
>
> >
> > Hi!
> >
> > We are using lucene in our project to search through information objects
> > which works fine. For indexing we use the StandardAnalyzer.
> > Now, we have to support the Chinese language. I found out that the
> Chinese
> > words and letters are correctly saved in the index but the query to
> search
> > for them does not work. Example: in English language the query is “text”
> > which we parse to “*text*”. If we search for Chinese words / phrases like
> > “佛山东方书城”the query is “*佛山东方书城*“ but there are no search results. If the
> > query places blanks between the single letters / symbols like this “*佛 山
> 东 方
> > 书 城*“ we are getting results. Does the StandardAnalyzer interpret each
> > Chinese letter as one word? What are best practices for this case? Shall
> we
> > use another analyzer (Chinese analyzer)? Or is it better to replace the
> > query parser in this case?
> >
> > Regards,
> > Jacqueline.
> >
>
>
>
> --
> Robert Muir
> rcmuir@gmail.com
>



-- 
Robert Muir
rcmuir@gmail.com

AW: Lucene and Chinese language

Posted by "Kolhoff, Jacqueline - ENCOWAY" <Ko...@encoway.de>.
How can I add this PositionFilter? I can't see anything in the API. I use lucene version 3.0.1, this is my query parser:

QueryParser queryParser = new QueryParser(Version.LUCENE_30, "myfieldname" , new StandardAnalyzer(Version.LUCENE_30));

-----Ursprüngliche Nachricht-----
Von: Robert Muir [mailto:rcmuir@gmail.com] 
Gesendet: Donnerstag, 1. Juli 2010 12:34
An: java-user@lucene.apache.org
Betreff: Re: Lucene and Chinese language

This is a bug in the queryparser. (
https://issues.apache.org/jira/browse/LUCENE-2458)

the problem has nothing to do with your choice of analyzer, it has to do
with how the query is formed.

Currently the queryparser uses a convoluted algorithm involving whitespace
(and not just the double quote operator as you would expect) to form phrase
queries. So, queries like this with no whitespace form phrase queries
always.

The only workaround for reasonably good results consists of two steps:
1. at query time (only!) add a
org.apache.lucene.analysis.position.PositionFilter (from contrib/analyzers)
to your analyzer. don't do this at index-time, just query-time!
2. this will make all terms in the query "synonyms" of each other to bypass
this problem, but will screw up scoring, so you might want to also extend
QueryParser in a custom way:

@Override
 protected BooleanQuery newBooleanQuery(boolean disableCoord) {
   // intentionally ignore disabled
   // coord() factor from the PositionFilter hack.
   return new BooleanQuery(false);
 }

2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>

>
> Hi!
>
> We are using lucene in our project to search through information objects
> which works fine. For indexing we use the StandardAnalyzer.
> Now, we have to support the Chinese language. I found out that the Chinese
> words and letters are correctly saved in the index but the query to search
> for them does not work. Example: in English language the query is “text”
> which we parse to “*text*”. If we search for Chinese words / phrases like
> “佛山东方书城”the query is “*佛山东方书城*“ but there are no search results. If the
> query places blanks between the single letters / symbols like this “*佛 山 东 方
> 书 城*“ we are getting results. Does the StandardAnalyzer interpret each
> Chinese letter as one word? What are best practices for this case? Shall we
> use another analyzer (Chinese analyzer)? Or is it better to replace the
> query parser in this case?
>
> Regards,
> Jacqueline.
>



-- 
Robert Muir
rcmuir@gmail.com

Re: Lucene and Chinese language

Posted by Robert Muir <rc...@gmail.com>.
This is a bug in the queryparser. (
https://issues.apache.org/jira/browse/LUCENE-2458)

the problem has nothing to do with your choice of analyzer, it has to do
with how the query is formed.

Currently the queryparser uses a convoluted algorithm involving whitespace
(and not just the double quote operator as you would expect) to form phrase
queries. So, queries like this with no whitespace form phrase queries
always.

The only workaround for reasonably good results consists of two steps:
1. at query time (only!) add a
org.apache.lucene.analysis.position.PositionFilter (from contrib/analyzers)
to your analyzer. don't do this at index-time, just query-time!
2. this will make all terms in the query "synonyms" of each other to bypass
this problem, but will screw up scoring, so you might want to also extend
QueryParser in a custom way:

@Override
 protected BooleanQuery newBooleanQuery(boolean disableCoord) {
   // intentionally ignore disabled
   // coord() factor from the PositionFilter hack.
   return new BooleanQuery(false);
 }

2010/7/1 Kolhoff, Jacqueline - ENCOWAY <Ko...@encoway.de>

>
> Hi!
>
> We are using lucene in our project to search through information objects
> which works fine. For indexing we use the StandardAnalyzer.
> Now, we have to support the Chinese language. I found out that the Chinese
> words and letters are correctly saved in the index but the query to search
> for them does not work. Example: in English language the query is “text”
> which we parse to “*text*”. If we search for Chinese words / phrases like
> “佛山东方书城”the query is “*佛山东方书城*“ but there are no search results. If the
> query places blanks between the single letters / symbols like this “*佛 山 东 方
> 书 城*“ we are getting results. Does the StandardAnalyzer interpret each
> Chinese letter as one word? What are best practices for this case? Shall we
> use another analyzer (Chinese analyzer)? Or is it better to replace the
> query parser in this case?
>
> Regards,
> Jacqueline.
>



-- 
Robert Muir
rcmuir@gmail.com