[jira] [Commented] (STANBOL-1128) Implement a Lucene FST based Entity Linking Engine (based on OpenSextant / SolrTextTagger)

["Rupert Westenthaler (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/STANBOL-1128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13715209#comment-13715209 ] 

Rupert Westenthaler commented on STANBOL-1128:
----------------------------------------------

Update of the current development status:

* The engine requires FST structures:
    * FST structures are read from within the SolrCore directory under "fst/{name}.{lang}.fst"
    * Runtime creation is optionally supported, but will require considerable CPU, memory and time

* The engine requires the SolrCore to be registered and available
    * NOTE that NO Solr queries are issued as linking in done in-memory!
    * However selected fields of the Lucene documents for linked Entities need to be read to obtain the labels, types and entity rankings

* Linking process:
    * I contributed the ability to only link selected tokens to the SolrTextTagger [1] project. This allows to have the same TextProcessing configurations as currently supported by the EntityLinkingEngine.
    * Detection of Entities with wrong Token order (e.g. {Family} {Given} for persons) can not be supported. Users will need to create those Label combinations at indexing time!
    * Linking is done based on the Solr Field configuration. This means that all available Solr Analyzers can be used for linking (even phonetic ...)

                
> Implement a Lucene FST based Entity Linking Engine (based on OpenSextant / SolrTextTagger)
> ------------------------------------------------------------------------------------------
>
>                 Key: STANBOL-1128
>                 URL: https://issues.apache.org/jira/browse/STANBOL-1128
>             Project: Stanbol
>          Issue Type: New Feature
>          Components: Enhancement Engines
>            Reporter: Rupert Westenthaler
>            Assignee: Rupert Westenthaler
>
> This will implement an in-memory EntityLinking EnhancementEngine based on Lucenes FST (Finite State Transducer) technology.
> This engine could make direct use of the Classes contained in the OpenSextant / SolrTextTagger code [1]. This uses a two layered FST 
>     (1) to represent words and 
>     (2) to map those words to phrases
> With this it is possible to efficiently hold big vocabularies in memory (> 300MByte for geonames.org). See the presentation at [2] for more details.
> While the license is fully compatible (ASL 2.0) the library is currently not available on Maven Central. We need to contact the author regarding this.
> [1] https://github.com/OpenSextant/SolrTextTagger
> [2] http://www.lucenerevolution.org/2013/Text-Tagging-with-Finite-State-Transducers

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira