You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Mordo, Aviran (EXP N-NANNATEK)" <av...@lmco.com> on 2007/05/03 22:06:04 UTC
Language detection library
Anyone knows of a good language detection library that can detect what
language a document (text) is ?
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Language detection library
Posted by Bob Carpenter <ca...@alias-i.com>.
>> Anyone knows of a good language detection library that can detect what
>> language a document (text) is ?
Language detection is easy. It's just a simple
text classification problem.
One way you can do this is using Lucene
itself. Create a so-called pseudo-document
for each language consisting of lots of text
(1 MB or more, ideally). Then build a Lucene
index using a character n-gram tokenizer.
Eg. "John Smith" tokenizes to "Jo", "oh",
"hn", "n ", " S", "Sm", "mi", "it", "th"
with 2-grams.
You'll have to make sure to index beyond the
first 1000 tokens or whatever Lucene is set to
by default.
To do language ID, just treat the language
to be identified as the basis of a query.
Parse it using the same character n-gram
tokenizer. The highest-scoring result is
the answer and if two score high, you know
there may be some ambiguity. You can't trust
Lucene's normalized scoring for rejection,
though.
Make sure the tokenizer includes spaces as
well as non-space characters (though all
spaces may be normalized to a single whitespace).
Using more orders (1-grams, 2-grams, 3-grams,
etc.) gives more accuracy; the IDF weighting
is quite sensible here and will work out the
details for the counts for you.
For a more sophisticated approach, check out
LingPipe's language ID tutorial, which is
based on probabilistic character language models.
Think of it as similar to the Lucene model but
with different term weighting.
http://www.alias-i.com/lingpipe/demos/tutorial/langid/read-me.html
Here's accuracy vs. input length on a set of 15
languages from the Leipzig Corpus collection (just
one of the many evals in the tutorial):
#chars accuracy
1 22.59%
2 34.82%
4 58.55%
8 81.17%
16 92.45%
32 97.33%
64 98.99%
128 99.67%
The end of the tutorial has references to other
popular language ID packages online (e.g. TextCat,
which is Gertjan van Noord's Perl package). And it
also has references to the technical background
on TF/IDF classification with n-grams and
character language models.
- Bob Carpenter
Alias-i
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Language detection library
Posted by karl wettin <ka...@gmail.com>.
4 maj 2007 kl. 02.20 skrev Chris Lu:
> I suppose if a document is indexed as English or French,
> when users searching the document,
> we need to parse the query as English or French also?
If you do some language specific token analysis such as stemming, yes.
Detecting the language on such small texts is sort of tricky though.
You might want to introduce more dimensions in the classifier: user
location, user locale, et c. Perhaps you want to store stemmed data
in language specific fields. It might also be a good idea to place an
initial query and re-classifiy to one of the top n scoring language
and then replace the query.
The easiest way out is to simply ask the user what language they want
to search in. And that seems to be the most common.
>
> --
> Chris Lu
> -------------------------
> Instant Scalable Full-Text Search On Any Database/Application
> site: http://www.dbsight.net
> demo: http://search.dbsight.com
> Lucene Database Search in 3 minutes:
> http://wiki.dbsight.com/index.php?
> title=Create_Lucene_Database_Search_in_3_minutes
>
>
> On 5/3/07, karl wettin <ka...@gmail.com> wrote:
>>
>> 3 maj 2007 kl. 22.06 skrev Mordo, Aviran (EXP N-NANNATEK):
>>
>> > Anyone knows of a good language detection library that can
>> detect what
>> > language a document (text) is ?
>>
>> I posted this some time back:
>>
>> https://issues.apache.org/jira/browse/LUCENE-826
>>
>> A bit of proof-of-concept:ish, but it does the job well if you ask
>> me. Uses Weka (GPL) and requires at least 150 characters to be
>> trusted.
>>
>>
>> --
>> karl
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Language detection library
Posted by Chris Lu <ch...@gmail.com>.
I suppose if a document is indexed as English or French,
when users searching the document,
we need to parse the query as English or French also?
--
Chris Lu
-------------------------
Instant Scalable Full-Text Search On Any Database/Application
site: http://www.dbsight.net
demo: http://search.dbsight.com
Lucene Database Search in 3 minutes:
http://wiki.dbsight.com/index.php?title=Create_Lucene_Database_Search_in_3_minutes
On 5/3/07, karl wettin <ka...@gmail.com> wrote:
>
> 3 maj 2007 kl. 22.06 skrev Mordo, Aviran (EXP N-NANNATEK):
>
> > Anyone knows of a good language detection library that can detect what
> > language a document (text) is ?
>
> I posted this some time back:
>
> https://issues.apache.org/jira/browse/LUCENE-826
>
> A bit of proof-of-concept:ish, but it does the job well if you ask
> me. Uses Weka (GPL) and requires at least 150 characters to be trusted.
>
>
> --
> karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Language detection library
Posted by karl wettin <ka...@gmail.com>.
3 maj 2007 kl. 22.06 skrev Mordo, Aviran (EXP N-NANNATEK):
> Anyone knows of a good language detection library that can detect what
> language a document (text) is ?
I posted this some time back:
https://issues.apache.org/jira/browse/LUCENE-826
A bit of proof-of-concept:ish, but it does the job well if you ask
me. Uses Weka (GPL) and requires at least 150 characters to be trusted.
--
karl
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org