You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@tika.apache.org by "Paul Ramirez (JIRA)" <ji...@apache.org> on 2015/07/23 19:39:04 UTC

[jira] [Created] (TIKA-1696) Language Identification with Text Processing Toolkit from MITLL

Paul Ramirez created TIKA-1696:
----------------------------------

             Summary: Language Identification with Text Processing Toolkit from MITLL
                 Key: TIKA-1696
                 URL: https://issues.apache.org/jira/browse/TIKA-1696
             Project: Tika
          Issue Type: New Feature
          Components: languageidentifier
            Reporter: Paul Ramirez
             Fix For: 1.10


The aim here is to extend the methods for language identification within text. MIT Lincoln Labs has an open source library [1] written in Julia. Having spoken  with the MITLL guys there is a possibility that there is a scala version of this library which would make it easier to package in with Tika. 

At this point I'm not quite sure how many languages this library supports by default but it can be extended when provided some training data.

[1] https://github.com/mit-nlp/Text.jl



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)