You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Eric Chow <er...@gmail.com> on 2005/01/21 10:49:18 UTC
Search Chinese in Unicode !!!
How to create index with chinese (in utf-8 encoding ) HTML and search
with Lucene ?
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Search Chinese in Unicode !!!
Posted by PA <pe...@gmail.com>.
On Jan 21, 2005, at 11:42, Eric Chow wrote:
> Search not really correct with UTF-8 !!!
Lucene works just fine with any flavor of Unicode as long as _your_
application knows how to consistently deal with Unicode as well.
Remember: the world is not just one Big5 pile.
As far as Analyzer goes, you may or may not be better off using
something more tailored to your linguistic needs. That said, even the
default Analyzer does a fairly decent job at handling non-roman
languages. YMMV.
Cheers
--
PA
http://alt.textdrive.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Search Chinese in Unicode !!!
Posted by René Hackl <re...@gmx.de>.
> > The modified code in SearchFiles.java:
> >
> >
> > BufferedReader in = new BufferedReader(new
> > InputStreamReader(System.in, "UTF-8"));
It might make sense to incorporate a similar change in WordlistLoader.
Instead of
freader = new FileReader(wordfile);
lnr = new LineNumberReader(freader);
I think it's preferable to do something like
LineNumberReader lnr = new LineNumberReader(new InputStreamReader(
new FileInputStream(wordfile), "UTF-8"));
to load even more languages' files, now that it resides in the
analysis-package.
Regards,
René
--
10 GB Mailbox, 100 FreeSMS http://www.gmx.net/de/go/topmail
+++ GMX - die erste Adresse für Mail, Message, More +++
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Search Chinese in Unicode !!!
Posted by Otis Gospodnetic <ot...@yahoo.com>.
I don't have a document with chinese characters to verify this, but it
looks right, so I'll add your change to SearchFiles.java.
Thanks,
Otis
--- Eric Chow <er...@gmail.com> wrote:
> Search not really correct with UTF-8 !!!
>
>
> The following is the search result that I used the SearchFiles in the
> lucene demo.
>
> d:\Downloads\Softwares\Apache\Lucene\lucene-1.4.3\src>java
> org.apache.lucene.demo.SearchFiles c:\temp\myindex
> Usage: java SearchFiles <idnex>
> Query: 經
> Searching for: g <<<<<<<<<<<<
> strange ??
> 3 total matching documents
> 0. ../docs/ChineseDemo.html <<<<<<<<<<<< this files
> contains the 經
> -
> 1. ../docs/luceneplan.html
> - Jakarta Lucene - Plan for enhancements to Lucene
> 2. ../docs/api/index-all.html
> - Index (Lucene 1.4.3 API)
> Query:
>
>
>
> From the above result only the ChineseDemo.html includes the
> character
> that I want to search !
>
>
>
>
> The modified code in SearchFiles.java:
>
>
> BufferedReader in = new BufferedReader(new
> InputStreamReader(System.in, "UTF-8"));
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Search Chinese in Unicode !!!
Posted by Eric Chow <er...@gmail.com>.
Search not really correct with UTF-8 !!!
The following is the search result that I used the SearchFiles in the
lucene demo.
d:\Downloads\Softwares\Apache\Lucene\lucene-1.4.3\src>java
org.apache.lucene.demo.SearchFiles c:\temp\myindex
Usage: java SearchFiles <idnex>
Query: 經
Searching for: g <<<<<<<<<<<< strange ??
3 total matching documents
0. ../docs/ChineseDemo.html <<<<<<<<<<<< this files contains the 經
-
1. ../docs/luceneplan.html
- Jakarta Lucene - Plan for enhancements to Lucene
2. ../docs/api/index-all.html
- Index (Lucene 1.4.3 API)
Query:
>From the above result only the ChineseDemo.html includes the character
that I want to search !
The modified code in SearchFiles.java:
BufferedReader in = new BufferedReader(new
InputStreamReader(System.in, "UTF-8"));
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Search Chinese in Unicode !!!
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jan 21, 2005, at 4:49 AM, Eric Chow wrote:
> How to create index with chinese (in utf-8 encoding ) HTML and search
> with Lucene ?
Indexing and searching Chinese basically is no different than using
English with Lucene. We covered a bit about it in Lucene in Action:
http://www.lucenebook.com/search?query=chinese
And a screenshot here:
http://www.blogscene.org/erik/LuceneInAction/i18n.html
The main issues of dealing with Chinese, and of course other languages,
are encoding concerns in both indexing and querying of reading in the
text and analysis (as you can see from the screenshot).
Lucene itself works with Unicode fine and you're free to index anything.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Search Chinese in Unicode !!!
Posted by 田春峰 <tc...@yahoo.com.cn>.
hi, Eric
If you can read chinese directly , Please reference to this blog:
http://blog.csdn.net/accesine960
or, search weblucene at www.sf.net which is a project based upon lucene by a chinese, name : chedong , his web site is : www.chedong.com
good luck
Eric Chow <er...@gmail.com> wrote:
How to create index with chinese (in utf-8 encoding ) HTML and search
with Lucene ?
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
自动签名：
请使用机器人服务:
msn机器人: xiaodingdong21@hotmail.com
qq机器人: 443803193
---------------------------------
Do You Yahoo!?
150万曲MP3疯狂搜，带您闯入音乐殿堂
美女明星应有尽有，搜遍美图、艳图和酷图
1G就是1000兆，雅虎电邮自助扩容！