You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Samphan Raruenrom (JIRA)" <ji...@apache.org> on 2006/03/01 11:45:39 UTC
[jira] Created: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
---------------------------------------------------------------
Key: LUCENE-503
URL: http://issues.apache.org/jira/browse/LUCENE-503
Project: Lucene - Java
Type: New Feature
Components: Analysis
Versions: 1.4
Reporter: Samphan Raruenrom
Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Samphan Raruenrom (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=comments#action_12413756 ]
Samphan Raruenrom commented on LUCENE-503:
------------------------------------------
All the code have been tested with Lucene 2.0.0.
Thanks Art for the info/URL. I've never known about Pichai's work before I started this project. However I heard about NECTEC's SansarnLook when I visit them and talk about my ThaiAnalyzer. My goal for this job is for the code to be included in Lucene for Thai to work out-of-the-box. So no more wheel reinventing.
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Attachments: TestThaiAnalyzer.java, ThaiAnalyzer.java, ThaiWordFilter.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Samphan Raruenrom (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=all ]
Samphan Raruenrom updated LUCENE-503:
-------------------------------------
Attachment: ThaiAnalyzer.java
ThaiAnalyzer which simply return a TokenFilter chain with ThaiWordFilter in the middle
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Attachments: ThaiAnalyzer.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Daniel Naber (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=comments#action_12377071 ]
Daniel Naber commented on LUCENE-503:
-------------------------------------
Thanks for your contribution. We're currently preparing Lucene 2.0 and as feature updates are only planned for the release after 2.0 it will take some more time to integrate this.
Two remarks:
-It uses the english stop words, does that make sense?
-Could you write some test cases, similar maybe to those for the French analyzer?
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Attachments: ThaiAnalyzer.java, ThaiWordFilter.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Samphan Raruenrom (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=comments#action_12374136 ]
Samphan Raruenrom commented on LUCENE-503:
------------------------------------------
I've changed the code to use java.text.BreakIterator instead of ICU4j to remove the dependency on ICU4j. The ThaiAnayzer is tested intensively by several groups of developers in at least two production systems (by To-Be-One Technology, who support the development) so it is quite stable. The code is rather small cause I try to make it as efficient and easy to read as possible. It's tested in Lucece 1.4 and lately in Lucene 1.9.1.
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Attachments: ThaiAnalyzer.java, ThaiWordFilter.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Resolved: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=all ]
Hoss Man resolved LUCENE-503:
-----------------------------
Resolution: Fixed
commited
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Assignee: Hoss Man
> Attachments: TestThaiAnalyzer.java, ThaiAnalyzer.java, ThaiWordFilter.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Samphan Raruenrom (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=all ]
Samphan Raruenrom updated LUCENE-503:
-------------------------------------
Attachment: TestThaiAnalyzer.java
Add TestThaiAnalyzer junit test, modified from TestFrenchAnalyzer. The Thai words are picked so that changing the dictionary (or algorithm in JDK) should not affect the test.
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Attachments: TestThaiAnalyzer.java, ThaiAnalyzer.java, ThaiWordFilter.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Updated: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Samphan Raruenrom (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=all ]
Samphan Raruenrom updated LUCENE-503:
-------------------------------------
Attachment: ThaiWordFilter.java
ThaiWordFilter which use java.text.BreakIterator to break Thai words into tokens
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Attachments: ThaiAnalyzer.java, ThaiWordFilter.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Samphan Raruenrom (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=comments#action_12377206 ]
Samphan Raruenrom commented on LUCENE-503:
------------------------------------------
> -It uses the english stop words, does that make sense?
Yes. Thai usually mix English words in Thai text here and there. So English stop words should apply but this is arguable. I'll consull with the developer community.
> -Could you write some test cases, similar maybe to those for the French analyzer?
OK. I'm thinking of writing them.
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Attachments: ThaiAnalyzer.java, ThaiWordFilter.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Commented: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Arthit Suriyawongkul (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=comments#action_12413695 ]
Arthit Suriyawongkul commented on LUCENE-503:
---------------------------------------------
related projects/implementations:
SansarnLook
based on Lucene, with additional ThaiAnalyzer
ref: http://sansarn.com/look/technique/
file: http://sansarn.com/look/download/
Pichai Ongvasith's ThaiAnalyzer
ref: http://mail-archives.apache.org/mod_mbox/lucene-java-dev/200402.mbox/%3C20040218004604.61360.qmail@web41501.mail.yahoo.com%3E
file: http://pichai.netfirms.com/thai_analyzer.zip
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Attachments: ThaiAnalyzer.java, ThaiWordFilter.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org
[jira] Assigned: (LUCENE-503) Contrib: ThaiAnalyzer to enable Thai
full-text search in Lucene
Posted by "Hoss Man (JIRA)" <ji...@apache.org>.
[ http://issues.apache.org/jira/browse/LUCENE-503?page=all ]
Hoss Man reassigned LUCENE-503:
-------------------------------
Assign To: Hoss Man
I don't know anything about the Thai language ... but this code is clean, fairly easy to follow, and has tests that pass.
If no one (who knows something about Thai) sees anything wrong with this implimentation and objects i'll commit it sometime this weekend.
> Contrib: ThaiAnalyzer to enable Thai full-text search in Lucene
> ---------------------------------------------------------------
>
> Key: LUCENE-503
> URL: http://issues.apache.org/jira/browse/LUCENE-503
> Project: Lucene - Java
> Type: New Feature
> Components: Analysis
> Versions: 1.4
> Reporter: Samphan Raruenrom
> Assignee: Hoss Man
> Attachments: TestThaiAnalyzer.java, ThaiAnalyzer.java, ThaiWordFilter.java
>
> Thai text don't have space between words. Usually, a dictionary-based algorithm is used to break string into words. For Lucene to be usable for Thai, an Analyzer that know how to break Thai words is needed.
> I've implemented such Analyzer, ThaiAnalyzer, using ICU4j DictionaryBasedBreakIterator for word breaking. I'll upload the code later.
> I'm normally a C++ programmer and very new to Java. Please review the code for any problem. One possible problem is that it requires ICU4j. I don't know whether this is OK.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org