svn commit: r1301374 - /incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/engines/keywordlinkingengine.mdtext

[rw...@apache.org]

Author: rwesten
Date: Fri Mar 16 08:25:56 2012
New Revision: 1301374

URL: http://svn.apache.org/viewvc?rev=1301374&view=rev
Log:
Formatting improvements, Updated links to the deprecated TaxonomyLinkingEngine

Modified:
    incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/engines/keywordlinkingengine.mdtext

Modified: incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/engines/keywordlinkingengine.mdtext
URL: http://svn.apache.org/viewvc/incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/engines/keywordlinkingengine.mdtext?rev=1301374&r1=1301373&r2=1301374&view=diff
==============================================================================
--- incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/engines/keywordlinkingengine.mdtext (original)
+++ incubator/stanbol/site/trunk/content/stanbol/docs/trunk/enhancer/engines/keywordlinkingengine.mdtext Fri Mar 16 08:25:56 2012
@@ -14,28 +14,28 @@ The example in the scene shows an config
 
 ### Configuration Parameter
 
-* __Name__(stanbol.enhancer.engine.name): The name of the Enhancement Engine. This name is used to refer an [EnhancementEngine](index.html) in [EnhancementChain](enhancementchain.html)s
-* __Referenced Site__(org.apache.stanbol.enhancer.engines.keywordextraction.referencedSiteId): The name of the ReferencedSite of the Stanbol Entityhub that holds the controlled vocabulary to be used for extracting Entities. "entityhub" or "local" can be used to extract Entities managed directly by the Entityhub.
-* __Label Field__(org.apache.stanbol.enhancer.engines.keywordextraction.nameField): The name of the property used to lookup Entities. Only a single field is supported for performance reasons. Users that want to use values of several fields should collect such values by an according configuration in the mappings.txt used during indexing. This [usage scenario](../../customvocabulary.html) provides more information on this.
-* __Case Sensitivity__(org.apache.stanbol.enhancer.engines.keywordextraction.caseSensitive): This allows to activate/deactivate case sensitive matching. It is important to understand that even with case sensitivity activated an Entity with the label such as "Anaconda" will be suggested for the mention of "anaconda" in the text. The main difference will be the confidence value of such a suggestion as with case sensitivity activated the starting letters "A" and "a" are NOT considered to be matching. See the second technical part for details about the matching process. Case Sensitivity is deactivated by default. It is recommended to be activated if controlled vocabularies contain abbreviations similar to commonly used words e.g. CAN for Canada.
-* __Type Field__(org.apache.stanbol.enhancer.engines.keywordextraction.typeField): Values of this field are used as values of the "fise:entity-types" property of created "fise:EntityAnnotation"s. The default is "rdf:type".
-* __Redirect Field__(org.apache.stanbol.enhancer.engines.keywordextraction.redirectField) and __Redirect Mode__(org.apache.stanbol.enhancer.engines.keywordextraction.redirectMode): Redirects allow to tell the KeywordLinkingEngine to follow a specific property in the knowledge base for matched entities. This feature e.g. allows to follow redirects from "USA" to "United States" as defined in Wikipedia. See "Processing of Entity Suggestions" for details. Possible valued for the Redirect Mode are "IGNORE" - deactivates this feature; "ADD_VALUES" - uses label, type informations of redirected entities, but keeps the URI of the extracted entity; "FOLLOW" - follows the redirect
-* __Min Token Length__(org.apache.stanbol.enhancer.engines.keywordextraction.minSearchTokenLength): While the KeywordLinkingEngine preferable uses POS (part-of-speach) taggers to determine if a word should matched with the controlled vocabulary the minimum token length provides a fall back if (a) no POS tagger is available for the language of the parsed text or (b) if the confidence of the POS tagger is lower than the threshold.
-* __Keyword Tokenizer__(org.apache.stanbol.enhancer.engines.keywordextraction.keywordTokenizer): This allows to use a special Tokenizer for matching keywords and alpha numeric IDs. Typical language specific Tokenizers tend to split such IDs in several tokens and therefore might prevent a correct matching. This Tokenizer should only be activated if the KeywordLinkingEngine is configured to match against IDs like ISBN numbers, Product IDs ... It should not be used to match against natural language labels. 
-* __Suggestions__(org.apache.stanbol.enhancer.engines.keywordextraction.maxSuggestions): The maximum number of suggested Entities.
-* __Languages__(org.apache.stanbol.enhancer.engines.keywordextraction.processedLanguages) and __Default Matching Language__(org.apache.stanbol.enhancer.engines.keywordextraction.defaultMatchingLanguage): The first allows to specify languages that should be processed by this engine. This is e.g. useful if the controlled vocabulary only contains labels in for a specific language but does not formally specify this information (by setting the "xml:lang" property for labels). The default matching language can be used to work around the exact opposite case. As an example in DBpedia labels do get the language of the dataset they are extracted from (e.g. all data extracted from en.wikipedia.org will get "xml:lang=en"). The default matching language allows to tell the KeywordLinkingEngine to use labels of that language for matching regardless of the language of the parsed content. In the case of DBpedia this allows e.g. to match persons mentioned in an Italian text with the english l
 abels extracted from en.wikipedia.org. Details about natural language processing features used by this engine are provided in the section "Multiple Language Support"
-* __Type Mappings__(org.apache.stanbol.enhancer.engines.keywordextraction.typeMappings): The FISE enhancement structure (as used by the Stanbol Enhancer) distinguishes TextAnnotations and EntityAnnotations. The Keyword linking engine needs to create both types of Annotations: TextAnnotations selecting the words that match some Entities in the Controlled Vocabulary and EntityAnnotations that represent an Entity suggested for a TextAnnotation. The Type Mappings are used to determine the "dc:type" of the TextAnnotation based on the types of the suggested Entity. The default configuration comes with mappings for Persons, Organizations, Places and Concepts but this fields allows to define additional mappings. For details see the section "Type Mapping Syntax" and "Processing of Entity Suggestions".
-* __Dereference Entities__(org.apache.stanbol.enhancer.engines.keywordextraction.dereference): If enabled this engine adds additional information about the suggested Entities to the Metadata of the enhanced content item.
-* __Ranking__(service.ranking): This property is used of two engines do use the same __Name__. In such cases the one with the higher ranking will be used to enhance content items. Typically users will not need to change this.
+* __Name__ _(stanbol.enhancer.engine.name)_: The name of the Enhancement Engine. This name is used to refer an [EnhancementEngine](index.html) in [EnhancementChain](enhancementchain.html)s
+* __Referenced Site__ _(org.apache.stanbol.enhancer.engines.keywordextraction.referencedSiteId)_: The name of the ReferencedSite of the Stanbol Entityhub that holds the controlled vocabulary to be used for extracting Entities. "entityhub" or "local" can be used to extract Entities managed directly by the Entityhub.
+* __Label Field__ _(org.apache.stanbol.enhancer.engines.keywordextraction.nameField)_: The name of the property used to lookup Entities. Only a single field is supported for performance reasons. Users that want to use values of several fields should collect such values by an according configuration in the mappings.txt used during indexing. This [usage scenario](../../customvocabulary.html) provides more information on this.
+* __Case Sensitivity__ _(org.apache.stanbol.enhancer.engines.keywordextraction.caseSensitive)_: This allows to activate/deactivate case sensitive matching. It is important to understand that even with case sensitivity activated an Entity with the label such as "Anaconda" will be suggested for the mention of "anaconda" in the text. The main difference will be the confidence value of such a suggestion as with case sensitivity activated the starting letters "A" and "a" are NOT considered to be matching. See the second technical part for details about the matching process. Case Sensitivity is deactivated by default. It is recommended to be activated if controlled vocabularies contain abbreviations similar to commonly used words e.g. CAN for Canada.
+* __Type Field__ _(org.apache.stanbol.enhancer.engines.keywordextraction.typeField)_: Values of this field are used as values of the "fise:entity-types" property of created "fise:EntityAnnotation"s. The default is "rdf:type".
+* __Redirect Field__ _(org.apache.stanbol.enhancer.engines.keywordextraction.redirectField)_ and __Redirect Mode__ _(org.apache.stanbol.enhancer.engines.keywordextraction.redirectMode)_: Redirects allow to tell the KeywordLinkingEngine to follow a specific property in the knowledge base for matched entities. This feature e.g. allows to follow redirects from "USA" to "United States" as defined in Wikipedia. See "Processing of Entity Suggestions" for details. Possible valued for the Redirect Mode are "IGNORE" - deactivates this feature; "ADD_VALUES" - uses label, type informations of redirected entities, but keeps the URI of the extracted entity; "FOLLOW" - follows the redirect
+* __Min Token Length__ _(org.apache.stanbol.enhancer.engines.keywordextraction.minSearchTokenLength)_: While the KeywordLinkingEngine preferable uses POS (part-of-speach) taggers to determine if a word should matched with the controlled vocabulary the minimum token length provides a fall back if (a) no POS tagger is available for the language of the parsed text or (b) if the confidence of the POS tagger is lower than the threshold.
+* __Keyword Tokenizer__ _(org.apache.stanbol.enhancer.engines.keywordextraction.keywordTokenizer)_: This allows to use a special Tokenizer for matching keywords and alpha numeric IDs. Typical language specific Tokenizers tend to split such IDs in several tokens and therefore might prevent a correct matching. This Tokenizer should only be activated if the KeywordLinkingEngine is configured to match against IDs like ISBN numbers, Product IDs ... It should not be used to match against natural language labels. 
+* __Suggestions__ _(org.apache.stanbol.enhancer.engines.keywordextraction.maxSuggestions)_: The maximum number of suggested Entities.
+* __Languages__ _(org.apache.stanbol.enhancer.engines.keywordextraction.processedLanguages)_ and __Default Matching Language__ _(org.apache.stanbol.enhancer.engines.keywordextraction.defaultMatchingLanguage)_: The first allows to specify languages that should be processed by this engine. This is e.g. useful if the controlled vocabulary only contains labels in for a specific language but does not formally specify this information (by setting the "xml:lang" property for labels). The default matching language can be used to work around the exact opposite case. As an example in DBpedia labels do get the language of the dataset they are extracted from (e.g. all data extracted from en.wikipedia.org will get "xml:lang=en"). The default matching language allows to tell the KeywordLinkingEngine to use labels of that language for matching regardless of the language of the parsed content. In the case of DBpedia this allows e.g. to match persons mentioned in an Italian text with the eng
 lish labels extracted from en.wikipedia.org. Details about natural language processing features used by this engine are provided in the section "Multiple Language Support"
+* __Type Mappings__ _(org.apache.stanbol.enhancer.engines.keywordextraction.typeMappings)_: The FISE enhancement structure (as used by the Stanbol Enhancer) distinguishes TextAnnotations and EntityAnnotations. The Keyword linking engine needs to create both types of Annotations: TextAnnotations selecting the words that match some Entities in the Controlled Vocabulary and EntityAnnotations that represent an Entity suggested for a TextAnnotation. The Type Mappings are used to determine the "dc:type" of the TextAnnotation based on the types of the suggested Entity. The default configuration comes with mappings for Persons, Organizations, Places and Concepts but this fields allows to define additional mappings. For details see the section "Type Mapping Syntax" and "Processing of Entity Suggestions".
+* __Dereference Entities__ _(org.apache.stanbol.enhancer.engines.keywordextraction.dereference)_: If enabled this engine adds additional information about the suggested Entities to the Metadata of the enhanced content item.
+* __Ranking__ _(service.ranking)_: This property is used of two engines do use the same __Name__. In such cases the one with the higher ranking will be used to enhance content items. Typically users will not need to change this.
 
 Additionally the following properties can be configured via a configuration file:
 
-* __Minimum Found Tokens__(org.apache.stanbol.enhancer.engines.keywordextraction.minFoundTokens): This allows to tell the KeywordLinking Engine how to deal with Entities that do not exactly match words in the text. Typical Examples are "George W. Bush" -> "George Walker Bush". This parameter allows the minimum number of tokens that need to match. The default value is '2'. Note that this does not apply for exact matches. Setting this to a high value can be used to force a mode that will only consider entities where all tokens of the label match the mention in the text.
-* __Minimum Pos Tag Probability__(org.apache.stanbol.enhancer.engines.keywordextraction.minPosTagProbability): The minimum probability of a POS (part-of-speech) tag. Tags with a lower probability will be ignored. In such cases the configured value for the __Min Token Length__ will apply. The value MUST BE in the range [0..1]
+* __Minimum Found Tokens__ _(org.apache.stanbol.enhancer.engines.keywordextraction.minFoundTokens)_: This allows to tell the KeywordLinking Engine how to deal with Entities that do not exactly match words in the text. Typical Examples are "George W. Bush" -> "George Walker Bush". This parameter allows the minimum number of tokens that need to match. The default value is '2'. Note that this does not apply for exact matches. Setting this to a high value can be used to force a mode that will only consider entities where all tokens of the label match the mention in the text.
+* __Minimum Pos Tag Probability__ _(org.apache.stanbol.enhancer.engines.keywordextraction.minPosTagProbability)_: The minimum probability of a POS (part-of-speech) tag. Tags with a lower probability will be ignored. In such cases the configured value for the __Min Token Length__ will apply. The value MUST BE in the range [0..1]
 
 ### Type Mappings Syntax
 
-The Type Mappings are used to determine the "dc:type" of the TextAnnotation based on the types of the suggested Entity. The field "Type Mappings" (property: org.apache.stanbol.enhancer.engines.keywordextraction.typeMappings) can be used to customize such mappings.
+The Type Mappings are used to determine the "dc:type" of the TextAnnotation based on the types of the suggested Entity. The field "Type Mappings" (property: _org.apache.stanbol.enhancer.engines.keywordextraction.typeMappings_) can be used to customize such mappings.
 
 This field uses the following syntax
 
@@ -151,6 +151,8 @@ Input Parameters
 * span: number of tokens selected by the current suggestion e.g. "Barack Hussein Obama" -> 2
 * label_tokens: number of tokens of the matched label of the current entity (label_token) e.g. "Barack Hussein Obama" -> 3
 
+The confidence is calculated as follows: 
+
     :::java
     confidence = (match/max_matched)^2 * (matched/span) * (matched/label_tokens)
 
@@ -162,16 +164,17 @@ Some Examples:
 
 The calculation of the confidence is currently direct part of the [EntityLinker](http://svn.apache.org/repos/asf/incubator/stanbol/trunk/enhancer/engines/keywordextraction/src/main/java/org/apache/stanbol/enhancer/engines/keywordextraction/linking/EntityLinker.java). To support different matching strategies this would need to be externalized into an own interface.
 
-## Future Plans for the TaxonomyLinkingEngine ##
-
-The TaxonomyLinkingEngine is still available and fully functional. However it is marked as deprecated and not included in any of the launchers. Current users are encouraged to switch over to the KeywordLinkingEngine. 
+## Notes about the TaxonomyLinkingEngine ##
 
-In the future it is planed to repurpose the TaxonomyLinkingEngine as a special version of the KeywordLinkingEngine with a specialized configuration and feature set targeted for (hierarchical) Taxonomies. 
+The KeywordLinkingEngine is a re-implementation of the TaxonomyLinkingEngine which is more modular and therefore better suited for future improvements and extensions as requested by [STANBOL-303](https://issues.apache.org/jira/browse/STANBOL-303). As of [STANBOL-506](https://issues.apache.org/jira/browse/STANBOL-506) this engine is now deprecated and will be deleted from the SVN.
 
-This will include: 
+<!--
+However there would be now the possibility to implement a new version of an TaxonomyLinkingEngine with special support for hierarchical taxonomies. Such an engine would feature:
 
-* default configuration specific for SKOS
-* support for term hierarchies - adding suggestions for parent concepts
+* default configuration optimized for SKOS
+* support for term hierarchies - adding suggestions for parent concepts. Optionally by using a transitive closure over the hierarchy.
+* support for SKOS matching relations
 * support for restricting enhancements to a specific Taxonomy (skos:ConceptScheme) - this would allow to index several taxonomies in the same ReferencedSite but still use only a specific one for the enhancements.
 
-
+One Idea would be to allow users to use [LDPath](http://code.google.com/p/ldpath/) to configure post processing rules applied to extracted concepts of the Taxonomy.
+-->