You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nutch.apache.org by "Jerome Charron (JIRA)" <ji...@apache.org> on 2005/05/26 00:37:57 UTC
[jira] Updated: (NUTCH-60) Bad language identifier plugin performances
[ http://issues.apache.org/jira/browse/NUTCH-60?page=all ]
Jerome Charron updated NUTCH-60:
--------------------------------
Attachment: NUTCH-60-050526.patch
Patch with some minor performances improvements, but with some configurations parameters that enable to improve performances.
See http://wiki.apache.org/nutch/LanguageIdentifierBenchs and http://wiki.apache.org/nutch/NewLanguageIdentifier (coming soon) for more details.
Shortly, it adds the following configuration parameters:
* lang.ngram.min.length : The minimum size of ngrams to uses to identify language (must be between 1 and lang.ngram.max.length). The larger is the range between lang.ngram.min.length and lang.ngram.max.length, the better is the identification, but the slowest it is.
* lang.ngram.max.length: The maximum size of ngrams to uses to identify language (must be between lang.ngram.min.length and 4). The larger is the range between lang.ngram.min.length and lang.ngram.max.length, the better is the identification, but the slowest it is.
* lang.analyze.max.length: The maximum bytes of data to uses to indentify the language (0 means full content analysis). The larger is this value, the better is the analyzis, but the slowest it is.
Some new ngram profiles have been generated for en, es, fr, nl, it, pt, da, sv, de, fi, el cause the new implementation need more ngrams in the profile, but it is backward compatible with old ones.
Some unitary tests added.
> Bad language identifier plugin performances
> -------------------------------------------
>
> Key: NUTCH-60
> URL: http://issues.apache.org/jira/browse/NUTCH-60
> Project: Nutch
> Type: Improvement
> Components: indexer
> Reporter: Jerome Charron
> Priority: Minor
> Attachments: NUTCH-60-050526.patch
>
> As reported by Stefan Groschupf (http://www.mail-archive.com/nutch-developers@lists.sourceforge.net/msg04090.html) the language identifier plugin consumes a lot of processing time.
> Some optimizations and/or configuration options are required.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
http://www.atlassian.com/software/jira