You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@stanbol.apache.org by "Rupert Westenthaler (JIRA)" <ji...@apache.org> on 2012/11/17 11:07:11 UTC

[jira] [Comment Edited] (STANBOL-740) Adopt the KeywordLinkingEngine to use the AnalyzedText content part

[ https://issues.apache.org/jira/browse/STANBOL-740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486009#comment-13486009 ]

Rupert Westenthaler edited comment on STANBOL-740 at 11/17/12 10:06 AM:
------------------------------------------------------------------------

With revision 1403242 a first implementation of the KeywordLinkingEngine that is based on the Stanbol NLP prodessing Module (STABOL-733) is available in the stanbol-nlp-processing branch [2]. This comment is intended to be moved to the documentation of the Stanbol Webpage as soon as this version is re-integrated to the trunk.

edit: brings the documentation in line with revision http://svn.apache.org/viewvc?rev=1410404&view=rev

## Configuration

Only changes to the current version

### Removed Features

* Keyword Tokenizer (org.apache.stanbol.enhancer.engines.keywordextraction.keywordTokenizer): This allowed to use a special Tokenizer for matching keywords and alpha numeric IDs. This feature is no longer available as they KeywordLinkingEngine does no longer the tokenizing of parsed texts and has therefore no influence on how the text is tokenized. To preserve this feature a new Engine that is specialised for this task needed to be implemented.

### Deprecated Configuration

* __Min POS Tag Probability__ _(org.apache.stanbol.enhancer.engines.keywordextraction.minPosTagProbability)_: While still functional users should use the "prob" param supported by the _Processed Languages_ configuration. See the own section for details.

### Updated Features

* __Min Matched Tokens__ _org.apache.stanbol.enhancer.engines.keywordextraction.minFoundTokens_ [1..*]::int: The minimum number of matching tokens. Only "matchable" tokens are counted. For full matches (where all tokens of the Label do match tokens in the text) this parameter is ignored.

This parameter is strongly related with the _Min Label Match Score_ Typical setting are

1. _Min Matched Tokens_=1 and _Min Label Match Score_ > 0.5 (e.g. 0.75)
2. _Min Matched Tokens_=2 and _Min Label Match Score_ <= 0.5 (e.g. 0.5)

For Labels containing of one or two words both options do have the same result, but for Longer labels (1) is more restrictive than (2). The important thing is that both options ensures that Labels with more than one tokens will not be considered if only a single token does match the text.

If used in combination with an disambiguation Engine one might want to consider to suggest Entities where only a single token of multi-token labels do match. In such cases a configuration like _Min Matched Tokens_=1 and _Min Label Match Score_ <= 0.5 (e.g. 0.4) might be considered. With such scenarios users will also want to considerable increase the value for _Max Suggestions_ (typically values > 10).

### New Features

* __Link ProperNouns only__ _(org.apache.stanbol.enhancer.engines.keywordextraction.properNounsState)_: This boolean switch allows easily to switch between linking all nouns (state=false) or only proper nouns (state=true). "Noun linking" is equivalent to the current behavior of the KeywordLinkingEngine while "ProperNoun linking" is more similar to using NER with the NamedEntityLinking engine. For linking against vocabularies that contain Entities typically mentioned in texts as ProperNouns activating this option will greatly improve performance as much less words need to be looked-up in the Vocabulary. When linking to a Vocabulary that defines Entities that might be mentioned as common nouns this option need to be deactivated.

* __Processed Languages__ _(org.apache.stanbol.enhancer.engines.keywordextraction.processedLanguages)_: This features allows (1) to explicitly define languages processed by the Engine and (2) allows to provide Language Specific Configurations. Language specific configurations will override/extend engine global configurations. See the next section for details

* __Max Search Token Distance__ _org.apache.stanbol.enhancer.engines.keywordextraction.maxSearchTokenDistance_ [1..*]::int: The maximum number of Tokens searched around linked Tokens to be included in searches within the linked vocabulary (default value is '3'). As an Example in the text section "at the University of Munich a new procedure to" only "Munich" would be looked-up in the Vocabulary in case "ProperNoun" linking is activated. However for searching possible matches in the Vocabulary it makes sense to use additional Tokens to reduce (and better rank) possible matches for for "Munich". Because of that "matchable" words surrounding looked-up tokens are considered to be included for searches in the Vocabulary. This parameter allows to configure the maximum distance of words that are used for such searches. Note that this parameter will not cause Words outside of a Chunk to be used for searches (unless "Ingore Chunks" option is activated).

* __Max Search Tokens__ _org.apache.stanbol.enhancer.engines.keywordextraction.maxSearchTokens_ [1..*]::int: The maximum number of Tokens used for searches in the Controlled Vocabulary (default value is '2'). This sets the maximum number of Tokens used in OR queries to the linked Vocabulary.

* _org.apache.stanbol.enhancer.engines.keywordextraction.lemmaMatching_ [true/false]: This allows to use the Lemma instead of the original token in the text for searching and matching Labels of Entities. Currently this feature is by default disabled

* __Min Label Match Score__ _org.apache.stanbol.enhancer.engines.keywordextraction.minLabelMatchFactor_ [0..1]::double: The "Label Score" [0..1] represents how much of the Label of an Entity matches with the Text. It compares the number of Tokens of the Label with the number of Tokens matched to the Text. Not exact matches for Tokens, or if the Tokens within the label do appear in an other order than in the text do also reduce this score. Entities are only considered if at least one of their labels cores higher than the minimum for all tree of _Min Label Match Score_, _Min Text Match Score_ and _Min Match Score_.

* __Min Text Match Score__ _org.apache.stanbol.enhancer.engines.keywordextraction.minTextMatchFactor_ [0..1]::double: The "Text Score" [0..1] represents how well the Label of an Entity matches to the selected Span in the Text. It compares the number of matched {@link Token} from the label with the number of Tokens enclosed by the Span in the Text an Entity is suggested for. Not exact matches for Tokens, or if the Tokens within the label do appear in an other order than in the text do also reduce this score. Entities are only considered if at least one of their labels cores higher than the minimum for all tree of _Min Label Match Score_, _Min Text Match Score_ and _Min Match Score_.

* __Min Match Score__ _org.apache.stanbol.enhancer.engines.keywordextraction.minTextMatchFactor_ [0..1]::double: Defined as the product of the "Text Score" with the "Label Score" - meaning that this value represents both how well the label matches the text and how much of the label is matched with the text. Entities are only considered if at least one of their labels cores higher than the minimum for all tree of _Min Label Match Score_, _Min Text Match Score_ and _Min Match Score_.

*_org.apache.stanbol.enhancer.engines.keywordextraction.dereferenceFields_: Allows to define additional fields that are included for dereferneced Entities. Only applied of "Dereference Entities" is enabled.

## Processed Language Configuration

With they key _'org.apache.stanbol.enhancer.engines.keywordextraction.processedLanguages'_ the processed language(s) and language specific configurations can be applied. The first usage of this configuration is to specify the languages processed by the engine and second to parse language specific how languages should be processed.

__1. Language Configuraiton:__

For the configuration of the processed languages the following syntax is used:

de
en

This would configure the Engine to only process German and English texts. It is also possible to explicitly exclude languages

!fr
!it
*

This specifies that all Languages other than French and Italien are processed.

Values MUST BE parsed as Array or Vector. This is done by using the ["elem1","elem2",...] syntax as defined by OSGI ".config" files. The following example shows the two above examples combined to a single configuration.

org.apache.stanbol.enhancer.engines.keywordextraction.processedLanguages=["!fr","!it","de","en","*"]

__2. Language specific Parameter__

In addition to specifying the processed languages this configuration can also be used to parse language specific parameters. The syntax for parameters is as follows

{language};{param-name}={param-value};{param-name}={param-value}
*;{param-name}={param-value};{param-name}={param-value}
;{param-name}={param-value};{param-name}={param-value}

The first line sets the parameter for {language}. The 2nd and 3rd line show that either the wildcard language '*' or the empty language '' can be used to configure parameters that are used as defaults for all languages.

### Language Processing Parameters

The following param-names are supported by the KeywordLinkingEngine

__Phrase level Parameters:__

* __pc__ {name}::LexicalCategory - The _Phrase Categories_ processed by the Engine. Valid values include the name's of members of the LexicalCategory enumeration (e.g. "Noun", "Verb", "Adjective", "Adposition", ...)
* __ptag__ {tag}::String - the _Phrase Tag_ processed by the Engine. This allows to configure the String tags as used by the Chunker of a Language. This should only be used of the Chunk types of the Chunker are not mapped with members of the LexicalCategory enumeration.
* __pprob__ [0..1)::double - the _Min Phrase Tag Probability_ for Chunks to be accepted as processable ('value/2' is sufficient for rejecting).
* __lmmtip__ [''/true/false]::boolean - the _Link Multiple Matchable Tokens in Phrases_ parameter. As the name says it allows to enable/disable the linking of multiple matchable tokens within the same Chunk. This is especially important if _Proper Noun Linking_ is active, as it allows to detect 'named entities' that are constituted by two common nouns. NOTE that 'lmmtip' is short for 'lmmtip=true'

__Token level Parameters:__

* __lc__ {name}::LexicalCategory - The linked _Token Categories_. Valid values include the name's of members of the LexicalCategory enumeration (e.g. "Noun", "Verb", "Adjective", "Adposition", …). Typical configurations include "lc=Noun" or an empty list ("lc" or "lc=") to deactivate all categories and provide more fine granular Pos or Tag level configuration.
* __pos__ {name}::Pos - This linked _Pos Types_. Valid values include the name's of members of the Pos enumeration (e.g. "ProperNoun", "CommonNoun", "Infinitive", "Gerund", "PresentParticiple" and ~150 others). This parameter can be used to provide a very fine granular configuration. It is e.g. used by the _Link ProperNouns only_ setting to define that only "pos=ProperNoun" are linked.
* __tag__ {tag}::String - The linked _Pos Tags_. This parameter allows to configure POS tags as used by the POS tagger. This is useful if those Tags are not mapped to LexicalCategories or Pos types.
*__prob__ [0..1)::double - the _Min PosTag Probability_. This parameter replaces the formally used _Min POS tag probability_ _(org.apache.stanbol.enhancer.engines.keywordextraction.minPosTagProbability)_ property. It defines the minimum confidence so that a POS annotation is accepted for linkable and matchable tokens ('value/2' is sufficient for rejecting none linked/matched tokens).
* __uc__ {NONE/MATCH/LINK}::string - the _Upper Case Token Mode_ allows to configure how upper case words are treated. There are three possible modes: (1) NONE: defines that they are not specially treated; (2) MATCH defines that they are considered as matchable tokens (independent of the POS tag or the token length; (3) LINK: defines that they are in any case linked with the vocabulary. The default is "LINK" - as upper case words often represent named entities - with the exception of German ('de') where the mode is set to MATCH - as all Nouns in German are upper case.

NOTE: that tokens are linked if any of "lc", "pos" or "tag" match the configuration. This means that adding "lc=Noun" will render "pos=ProperNoun" useless as the Pos type ProperNoun is already included in the LexicalCategory Noun.

__Examples:__

The default configuration for the KeywordLinkingEngine uses the following setting

*;lmmtip;uc=LINK;prop=0.75;pprob=0.75
de;uc=MATCH
es;lc=Noun
nl;lc=Noun

The first line enable _Link Multiple Matchable Tokens in Phrases_ and linking of upper case tokens for all languages. In addition it sets the minimum probabilities for Pos- and Phrase annotations to 0.75 (what would be also the default). The following three lines provide additional language specific defaults. For German the upper case mode is reset to MATCH as in German all Nouns use upper case. For Spain and Dutch linking for the LexicalCategory Noun is enabled. This is because the OpenNLP POS tagger for those languages does not support ProperNoun's and therefore the Engine would not link any tokens if _Link ProperNouns only_ is enabled. The same configuration in the OSGI '.config' file syntax would look like follows

org.apache.stanbol.enhancer.engines.keywordextraction.processedLanguages=["*;lmmtip;uc\=LINK;prop\=0.75;pprob\=0.75","de;uc\=MATCH","es;lc\=Noun","nl;lc\=Noun"]

The 2nd example shows how to define default settings without using the wildcard '*' that would enable processing of all languages. The following example shows an configuration that only enables English and ignores text in all other languages.

;lmmtip;uc=LINK;prop=0.75;pprob=0.75
en
de;uc=MATCH

## Extension Points

This section describes Interfaces that are used as Extension Points by the KeywordLinkingEngine

### EntitySearcher

The EntitySearch Interface is used by the KeywordLinkingEngine to search for Entities in the linked Vocabulary. Currently the StanbolEntityhub based implementations are instantiated based on the value of the _'org.apache.stanbol.enhancer.engines.keywordextraction.referencedSiteId'_. Users that want to use a different implementation of this Interface to be used for linking will need to extend the KeywordLinkingEngine and override the #activateEntitySearcher(ComponentContext context, Dictionary<String,Object> configuration) and #deactivateEntitySearcher(). Those methods are called during activation/deactivation of the KeywordLinkingEngine and are expected to set/unset the #entitySearcher field.

### LabelTokenizer

The LabelTokenizer interface is used to tokenize labels of Entities from the linked Vocabulary. As the matching process of the KeywordLinkingEngine is based on Tokens (words) multi-word labels (e.g. Univerity of Munich) need to be tokenized before they can be matched against the current context in the Text.

LabelTokenizer are OSGI services. Their configuration optionally can define the _'enhancer.engines.keywordextraction.labeltokenizer.languages'_ property. Values are considered to be language configurations. Configurations can explicitly include/exclude languages. Also a wildcard is supported (e.g. "en,de" would include English and German; "!it,!fr,*" would specify all languages expect Italian and French. If no configuration is provided than "*" (all languages) is assumed.

The KeywordLinkingEngine will - by default - always use the LabelTokenizer with the highest "service.ranking" for a given language to tokenize labels. By default it comes with an OpenNLP based Tokenizer implementation that registers itself for all languages with a "service.ranking" of "-1000".

Users that want to use a different Tokenizer need to register an implementation for the given language(s) with an higher "service.ranking". Users that want to provide there own LabelTokenizer and ignore the values provided by OSGI need to extend the KeywordLinkingEngine set the #labelTokenizer field themself AND override the #bindLabelTokenizer(LabelTokenizerManager ltm) and #unbindLabelTokenizer(LabelTokenizerManager ltm) methods in a way that they do NOT change the #labelTokenizer field.

[1] http://svn.apache.org/viewvc?rev=1403242&view=rev
[2] https://svn.apache.org/repos/asf/stanbol/branches/stanbol-nlp-processing

was (Author: rwesten):
With revision 1403242 a first implementation of the KeywordLinkingEngine that is based on the Stanbol NLP prodessing Module (STABOL-733) is available in the stanbol-nlp-processing branch [2]. This comment is intended to be moved to the documentation of the Stanbol Webpage as soon as this version is re-integrated to the trunk.

## Configuration

Only changes to the current version

### Removed Features

### New Features

* _org.apache.stanbol.enhancer.engines.keywordextraction.maxSearchTokenDistance_: The maximum number of Tokens searched around linked Tokens to be included in searches within the linked vocabulary (default value is '3'). As an Example in the text section "at the University of Munich a new procedure to" only "Munich" would be looked-up in the Vocabulary in case "ProperNoun" linking is activated. However for searching possible matches in the Vocabulary it makes sense to use additional Tokens to reduce (and better rank) possible matches for for "Munich". Because of that "matchable" words surrounding looked-up tokens are considered to be included for searches in the Vocabulary. This parameter allows to configure the maximum distance of words that are used for such searches. Note that this parameter will not cause Words outside of a Chunk to be used for searches (unless "Ingore Chunks" option is activated).

* _org.apache.stanbol.enhancer.engines.keywordextraction.masSearchTokens_: The maximum number of Tokens used for searches in the Controlled Vocabulary (default value is '2'). This sets the maximum number of Tokens used in OR queries to the linked Vocabulary.

### Processed Language Configuration

With they key _'org.apache.stanbol.enhancer.engines.keywordextraction.processedLanguages'_ the processed language(s) and language specific configurations can be applied.

For the configuration of the processed languages the following syntax is used:

de
en

This would configure the Engine to only process German and English texts. It is also possible to explicitly exclude languages

!fr
!it
*
This specifies that all Languages other than French and Italien are processed.

org.apache.stanbol.enhancer.engines.keywordextraction.processedLanguages=["!fr","!it","de","en","*"]

In addition to specifying the processed languages this configuration can also be used to parse language specific parameters. The syntax for parameters is as follows

{language};{param-name}={param-value};{param-name}={param-value}

The following param-names are supported by the KeywordLinkingEngine

* __lc__: This allows to parse LexicalCategories of words that shall be looked up in the Vocabulary. Valid values include the name's of members of the LexicalCategory enumeration (e.g. "Noun", "Verb", "Adjective", "Adposition", ...)
* __pos__: This allows to parse Pos types of words that shall be looked up in the Vocabulary. Valid values include the name's of members of the Pos enumeration (e.g. "ProperNoun", "CommonNoun", "Infinitive", "Gerund", "PresentParticiple" and ~150 others).
* __tag__: This allows to parse string tags used by the POS tagger for an language. Words that use those tags will be lokked-up with the vocabulary.
*__prob__: Allows a language specific setting of the _Min POS tag probability_ _(org.apache.stanbol.enhancer.engines.keywordextraction.minPosTagProbability)_. This value [0..1] is used to decide if a POS annotation is confident enough to use it for linking or rejecting ('value/2' is sufficient for rejecting).

Note that a word is linked if either "lc", "pos" or "tag" do match. So setting "pos=ProperNoun" will not have any effect if "lc=Noun" is already defined.

The following shows an "Processed Language Configuration" using all of the above mentioned features

!fr
!it
nl;lc=Noun
*;pos=ProperNoun

This would process all languages other that French and Italien; link all Nouns for Dutch texts and only ProperNouns for all others.

Users that want to define default parameters without using the "*" - wildcard language can use an empty language for parsing the parameters. Here an example

nl;lc=Noun
da
en
es
pt
sv
de
;pos=ProperNoun

This explicitly includes the seven languages for that OpenNLP POS models are included in the Stanbol Full Launcher. In addition it sets "Noun" linking for Dutch - as the POS tagset for this language does not distinguish between ProperNouns and CommonNouns. For the other six languages only "ProperNouns" are linked.

Users that directly parse configurations as OSGI ".config" need to properly escape configured parameters. The following example shows the above configuration in the syntax used by ".config" files

org.apache.stanbol.enhancer.engines.keywordextraction.processedLanguages=["nl;lc\=Noun","da","en","es","pt","sv","de",";pos\=ProperNoun"]

## Extension Points

This section describes Interfaces that are used as Extension Points by the KeywordLinkingEngine

### EntitySearcher

### LabelTokenizer

[1] http://svn.apache.org/viewvc?rev=1403242&view=rev
[2] https://svn.apache.org/repos/asf/stanbol/branches/stanbol-nlp-processing

> Adopt the KeywordLinkingEngine to use the AnalyzedText content part
> -------------------------------------------------------------------
>
> Key: STANBOL-740
> URL: https://issues.apache.org/jira/browse/STANBOL-740
> Project: Stanbol
> Issue Type: Sub-task
> Reporter: Rupert Westenthaler
> Assignee: Rupert Westenthaler
>
> The KeywordLinkingEngine currently does both NLP processing AND linking against the target vocabulary. Up to now this was the only possibility as separating those two things was not feasible with the limitations of the RDF metadata.
> With the introduction of the AnalyzedText content part the NLP processing part needs no longer be part of the KeywordLinkingEngine.
> This issue covers
> * removal of the NLP related functionality from the KeywordLinkingEngine
> * reimplementation of the linking part on top of the API provided by the AnalyzedText contentpart
> * add support fore new features of the NLP chain
> * use lemmas - if available - for entity lookup
> * use POS tagset mappings to the OLIA ontology to decide what tokens to lookup
> After this change the KeywordLinkingEngine will also be able to work in combination with any NLP framework that is integrated with the Stanbol NLP components (writes its data to the AnalyzedText content part).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira