You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uima.apache.org by pk...@apache.org on 2012/11/21 18:01:02 UTC

svn commit: r1412195 - /uima/sandbox/trunk/TextMarker/uima-docbook-textmarker/src/docbook/tools.textmarker.overview.xml

Author: pkluegl
Date: Wed Nov 21 17:01:02 2012
New Revision: 1412195

URL: http://svn.apache.org/viewvc?rev=1412195&view=rev
Log:
UIMA-2285
- removed old/incorrect overview
- added program listings for examples
- added some temporary notes for new chapter

Modified:
    uima/sandbox/trunk/TextMarker/uima-docbook-textmarker/src/docbook/tools.textmarker.overview.xml

Modified: uima/sandbox/trunk/TextMarker/uima-docbook-textmarker/src/docbook/tools.textmarker.overview.xml
URL: http://svn.apache.org/viewvc/uima/sandbox/trunk/TextMarker/uima-docbook-textmarker/src/docbook/tools.textmarker.overview.xml?rev=1412195&r1=1412194&r2=1412195&view=diff
==============================================================================
--- uima/sandbox/trunk/TextMarker/uima-docbook-textmarker/src/docbook/tools.textmarker.overview.xml (original)
+++ uima/sandbox/trunk/TextMarker/uima-docbook-textmarker/src/docbook/tools.textmarker.overview.xml Wed Nov 21 17:01:02 2012
@@ -23,180 +23,140 @@ KIND, either express or implied.  See th
 specific language governing permissions and limitations
 under the License.
 -->
-<chapter id="ugr.tools.tm.introduction">
-  <title>TextMarker</title>
-  <para>The TextMarker system is an open source tool for the development of rule-based information
-    extraction applications. The development environment is based on the DLTK framework. It supports
-    the knowledge engineer with a full-featured rule editor, components for the explanation of the
-    rule inference and a build process for generic UIMA Analysis Engines and Type Systems. Therefore
-    TextMarker components can be easily created and combined with other UIMA components in different
-    information extraction pipelines rather flexibly. TextMarker applies a specialized rule
-    representation language for the effective knowledge formalization: The rules of the TextMarker
-    language are composed of a list of rule elements that themselves consists of four parts: The
-    mandatory matching condition establishes a connection to the input document by referring to an
-    already existing concept, respectively annotation. The optional quantifier defines the usage of
-    the matching condition similar to regular expressions. Then, additional conditions add
-    constraints to the matched text fragment and additional actions determine the consequences of
-    the rule. Therefore, TextMarker rules match on a pattern of given annotations and, if the
-    additional conditions evaluate true, then they execute their actions, e.g. create a new
-    annotation. If no initial annotations exist, for example, created by another component, a
-    scanner is used to seed simple token annotations contained in a taxonomy. The TextMarker system
-    provides unique functionality that is usually not found in similar systems. The actions are able
-    to modify the document either by replacing or deleting text fragments or by filtering the view
-    on the document. In this case, the rules ignore some annotations, e.g. HTML markup, or are
-    executed only on the remaining text passages. The knowledge engineer is able to add heuristic
-    knowledge by using scoring rules. Additionally, several language elements common to scripting
-    languages like conditioned statements, loops, procedures, recursion, variables and expressions
-    increase the expressiveness of the language. Rules are able to directly invoke external rule
-    sets or arbitrary UIMA Analysis Engines and foreign libraries can be integrated with the
-    extension mechanism for new language elements.
-
+<chapter id="ugr.tools.tm.overview">
+  <title>TextMarker Overview</title>
+  <para>
+   
   </para>
-  <section id="ugr.tools.tm.introduction.metaphor">
-    <title>Introduction</title>
-    <para> In manual information extraction humans often apply a strategy according to a highlighter
-      metaphor: First relevant headlines are considered and classified according to their content by
-      coloring them with different highlighters. The paragraphs of the annotated headlines are then
-      considered further. Relevant text fragments or single words in the context of that headline
-      can then be colored. In this way, a top-down analysis and extraction strategy is implemented.
-      Necessary additional information can then be added that either refers to other text segments
-      or contains valuable domain specific information. Finally the colored text can be easily
-      analyzed concerning the relevant information.The TextMarker system (textmarker is a common
-      german word for a highlighter) tries to imitate this manual extraction method by formalizing
-      the appropriate actions using matching rules: The rules mark sequences of words, extract text
-      segments or modify the input document depending on textual features.The default input for the
-      TextMarker system is semi-structured text, but it can also process structured or free text.
-      Technically, HTML is often the input format, since most word processing documents can be
-      converted to HTML. Additionally, the TextMarker systems offers the possibility to create a
-      modified output document.
+  <section id="ugr.tools.tm.overview.intro">
+    <title>What is TextMarker?</title>
+    <para>
+      - rule-based script language
+      - imperative execution
+      - extensive tooling support, writing rules is tedious, needs support
+      - all about UIMA, AE, TypeSystem... 
     </para>
   </section>
-  <section id="ugr.tools.tm.introduction.concepts">
-    <title>Core Concepts</title>
+  
+  <section id="ugr.tools.tm.overview.gettingstarted">
+    <title>Getting started</title>
     <para>
-      As a first step in the extraction process the TextMarker system uses a tokenizer (scanner) to
-      tokenize the input document and to create a stream of basic symbols. The types and valid
-      annotations of the possible tokens are predefined by a taxonomy of annotation types.
-      Annotations simply refer to a section of the input document and assign a type or concept to
-      the respective text fragment. The figure on the right shows an excerpt of a basic annotation
-      taxonomy: CW describes all tokens, for example, that contains a single word starting with a
-      capital letter, MARKUP corresponds to HTML or XML tags, and PM refers to all kinds of
-      punctuations marks. Take a look at [basic annotations|BasicAnnotationList] for a complete list
-      of initial annotations.
-      <screenshot>
-        <mediaobject>
-          <imageobject>
-            <imagedata scale="80" format="PNG" fileref="&imgroot;symboltaxo.png" />
-          </imageobject>
-          <textobject>
-            <phrase>Part of a taxonomy for basic annotation types.</phrase>
-          </textobject>
-        </mediaobject>
-      </screenshot>
-      By using (and extending) the taxonomy, the knowledge engineer is able to choose the most
-      adequate types and concepts when defining new matching rules, i.e., TextMarker rules for
-      matching a text fragment given by a set of symbols to an annotation. If the capitalization of
-      a word, for example, is of no importance, then the annotation type W that describes words of
-      any kind can be used. The initial scanner creates a set of basic annotations that may be used
-      by the matching rules of the TextMarker language. However, most information extraction
-      applications require domain specific concepts and annotations. Therefore, the knowledge
-      engineer is able to extend the set of annotations, and to define new annotation types tuned to
-      the requirements of the given domain. These types can be flexibly integrated in the taxonomy
-      of annotation types. One of the goals in developing a new information extraction language was
-      to maintain an easily readable syntax while still providing a scalable expressiveness of the
-      language. Basically, the TextMarker language contains expressions for the definition of new
-      annotation types and for defining new matching rules. The rules are defined by a list of rule
-      elements. Each rule element contains at least a basic matching condition referring to text
-      fragments or already specified annotations. Additionally a list of conditions and actions may
-      be specified for a rule element. Whereas the conditions describe necessary attributes of the
-      matched text fragment, the actions point to operations and assignments on the current
-      fragments. These actions will then only be executed if all basic conditions matched on a text
-      fragment or the annotation and the related conditions are fulfilled.
+      - know about the core concepts of UIMA, this documentation assumes that the reader knows about that, CAS, Feature Structure, Annotation, Type, Analysis Engine, Type System,...
+      - recommended way to learn textmarker? how to read this documentation?
+      - installation of the IDE
+      - download the examples and play around
+      - read about the core concepts of TextMarker and take a look at the language examples
+      - read the chapter about language for a precise description of the language, rather a reference book
+      - workbench chapter for documentation of eclipse-based tooling
     </para>
   </section>
-  <section id="ugr.tools.tm.introduction.examples">
-    <title>Examples</title>
+  
+  <section id="ugr.tools.tm.overview.coreconcepts">
+    <title>Core Concepts</title>
     <para>
-      The usage of the language and its readability can be demonstrated by simple examples:
-      <programlisting><![CDATA[CW{INLIST('animals.txt') -> MARK(Animal)};        Animal "and" Animal{-> MARK(Animalpair, 1, 2, 3)}; ]]></programlisting>
-      The first rule looks at all capitalized words that are listed in an external document
-      animals.txt and creates a new annotation of the type animal using the boundaries of the
-      matched word. The second rule searches for an annotation of the type animal followed by the
-      literal and and a second animal annotation. Then it will create a new annotation animalpair
-      covering the text segment that matched the three rule elements (the digit parameters refer to
-      the number of matched rule element).
-      <programlisting><![CDATA[Document{-> MARKFAST(Firstname, 'firstnames.txt')};        Firstname CW{-> MARK(Lastname)};        Paragraph{VOTE(Firstname, Lastname) -> LOG("Found more Firstnames than Lastnames")}; ]]></programlisting>
-      In this example, the first rule annotates all words that occur in the external document
-      firstnames.txt with the type firstname. The second rule creates a lastname annotation for all
-      capitalized word that follow a firstname annotation. The last rule finally processes all
-      paragraph} annotations. If the VOTE condition counts more firstname than lastname annotations,
-      then the rule writes a log entry with a predefined message.
-      <programlisting><![CDATA[ANY+{PARTOF(Paragraph), CONTAINS(Delete, 50, 100, true) -> MARK(Delete)};        Firstname{-> MARK(Delete,1 , 2)} Lastname;     Delete{-> DEL};   ]]></programlisting>
-      Here, the first rule looks for sequences of any kind of tokens except markup and creates one
-      annotation of the type delete for each sequence, if the tokens are part of a paragraph
-      annotation and contains together already more than 50% of delete annoations. The + signs
-      indicate this greedy processing. The second rule annotates first names followed by last names
-      with the type delete and the third rule simply deletes all text segments that are associated
-      with that delete annotation.
+      - what does it look like? rule composition
+      - imperative execution, advantages and disadvantages
+      - depth first matching
+      - distinctive language elements for some tasks, one action or condition for one task
+      - where to start with matching? annotations in CAS, default annotation seeding 
+      - work with annotations, just spam them
+      - different approach to get to the final annotation, bottom-up, top-down, transformation-based
+      - scripting elements, strict rule-based approach sometimes inconvenient
+      - seamless integration with other uima components
+      - special features: scoring rules, filtering, modification
     </para>
   </section>
-  <section id="ugr.tools.tm.introduction.features">
-    <title>Special Features</title>
-    <para> The TextMarker language features some special characteristics that are usually not found
-      in other rule-based information extraction systems or even shift it towards scripting
-      languages. The possibility of creating new annotation types and integrating them into the
-      taxonomy facilitates an even more modular development of information extraction systems. Read
-      more about robust extraction using filtering, complex control structures and heuristic
-      extraction using scoring rules.
+  
+  <section id="ugr.tools.tm.overview.examples">
+    <title>Learning by Example</title>
+    <para>
+      TODO add simple introduction to a valid script file.
     </para>
-  </section>
-  <section id="ugr.tools.tm.introduction.getstarted">
-    <title>Get started</title>
-    <para> This section page gives you a short, technical introduction on how to get started with
-      TextMarker system and mostly just links the information of the other wiki pages. Some
-      knowledge about the usage of Eclipse and central concepts of UIMA are useful. TextMarker
-      consists of the TextMarker rule language (and of course the rule inference) and the TextMarker
-      workbench. Additionally, the CEV plugin is used to edit and visualize annotated text. The
-      TextRuler system with implementations of well known rule learning methods and development
-      extension with support for test-driven development are already integrated.
+    <programlisting><![CDATA[DECLARE Animal;
+CW{REGEXP("dog") -> MARK(Animal)};]]></programlisting>
+
+    <programlisting><![CDATA[WORDLIST AnimalsList = 'Animals.txt;'
+DECLARE Animal;
+Document{-> MARKFAST(Animal, AnimalsList)};
+]]></programlisting>
+    
+    <programlisting><![CDATA[DECLARE AnimalEnum;
+(Animal COMMA)+{-> MARK(AnimalEnum,1,2)} Animal;]]></programlisting>
+
+    <programlisting><![CDATA[(Animal (COMMA | SEMICOLON))+{-> MARK(AnimalEnum,1,2)} Animal;]]></programlisting>
+
+
+    <programlisting><![CDATA[DECLARE Headline;
+Paragraph{CONTAINS(Bold, 90, 100, true), 
+    CONTAINS(Underlined, 90, 100, true), ENDSWITH(COLON) 
+    -> MARK(Headline)};]]></programlisting>
+    
+    <programlisting><![CDATA[Headline{-CONTAINS(W) -> UNMARK(Headline)};]]></programlisting>
+    
+    <programlisting><![CDATA[]]>Headline{-> EXPAND(Headline, 1, 2)} COLON;</programlisting>
+    
+    <programlisting><![CDATA[DECLARE Month, Year, Date;
+ANY{INLIST(MonthsList) -> MARK(Month), MARK(Date,1,3)}
+    PERIOD? NUM{REGEXP(".{2,4}") -> MARK(Year))};]]></programlisting>
+
+    <programlisting><![CDATA[DECLARE Text1, Text2;
+ANY+{ -> MARK(Text1)};
+ANY+{-PARTOF(Text2) -> MARK(Text2)};]]></programlisting>
+
+    
+    <programlisting><![CDATA[DECLARE Annotation EmplRelation 
+    (Employee employeeRef, Employer employerRef);
+Sentence{CONTAINS(EmploymentIndicator) -> CREATE(EmplRelation,
+    "employeeRef" = Employee, "employerRef" = Employer)};]]></programlisting>
+
+    <programlisting><![CDATA[DECLARE Annotation MoneyAmount(STRING currency, INT amount);
+INT moneyAmount;
+STRING moneyCurrency;
+NUM{PARSE(moneyAmount)} SPECIAL{REGEXP("€") -> MATCHEDTEXT(moneyCurrency), 
+    CREATE(MoneyAmount, 1, 2, "amount" = moneyAmount, 
+        "currency" = moneyCurrency)};]]></programlisting>
+
+    <programlisting><![CDATA[SCRIPT uima.textmarker.example.SecondaryScript;
+Document{-> CALL(SecondaryScript)};]]></programlisting>
+
+    <programlisting><![CDATA[TYPESYSTEM my.package.NamedEntityTypeSystem;
+Person{PARTOF(Organization) -> UNMARK(Person)};
+]]></programlisting>
+
+
+    <programlisting><![CDATA[BLOCK(English) Document{FEATURE("language", "en")} {
+    // rules for english documents
+}
+BLOCK(English) Document{FEATURE("language", "dn")} {
+    // rules for german documents
+}]]></programlisting>
+
+    <programlisting><![CDATA[DECLARE SentenceWithNoLeadingNP;
+BLOCK(ForEach) Sentence{} {
+    Document{-STARTSWITH(NP) -> MARK(SentenceWithNoLeadingNP)};
+}
+]]></programlisting>
+    
+    <programlisting><![CDATA[ENGINE utils.Modifier;
+Date{-> DEL};
+MoneyAmount{-> REPLACE("<MoneyAmount/>")};
+Document{-> COLOR(Headline, "green")};
+Document{-> EXEC(Modifier)};
+]]></programlisting>
+    
+    <para>
+      - example with default filtering
     </para>
-    <section id="ugr.tools.tm.introduction.getstarted.running">
-      <title>Up and running</title>
-      <para> First of all, install the Workbench and read the introduction and its examples. In
-        order to verify if the Workbench is correctly installed, take a look at Help-About
-        Eclipse-Installation Details and compare the installed plugins with the plugins you copied
-        into the plugins folder of your Eclipse application. Normally most of the plugins do not
-        cause any troubles, but the CEV does because of the XPCom and XULRunner dependencies. You
-        should at least get the XPCom plugin up and running. However, you cannot use the additional
-        HTML functionality without the XULRunner plugin. If the plugins of the installation guide do
-        not work properly and a google search for a suiteable plugin is not successful, then write a
-        mail to the user list and we will try to solve the problem. If all plugins are correctly
-        installed, then start the Eclipse application and switch to the TextMarker perspective
-        (Window-Open Perspective-Other...)
-      </para>
-    </section>
-    <section id="ugr.tools.tm.introduction.getstarted.example">
-      <title>Learn by example</title>
-      <para> Having a running Workbench download the example project and import/copy this TextMarker
-        project into your workspace. The project contains some simple rules for extraction the
-        author, title and year of reference strings. Next, take a look at the project structure and
-        the syntax and compare it with the example project and its contents. Open the Main.tm
-        TextMarker script in the folder script/de.uniwue.example and press the Run button in the
-        Eclipse toolbar. The docments in the input folder will then be processed by the Main.tm file
-        and the result of the information extraction task is placed in the output folder. As you can
-        see, there are four files: an xmiCAS for each input file and a HTML file (the
-        modifed/colored result). Open one of the .xmi files with the CAS Editor plugin (-popup
-        menu-Open with) and select some checkboxes in the Annotation Browser view.
-      </para>
-    </section>
-    <section id="ugr.tools.tm.introduction.getstarted.doit">
-      <title>Do it yourself</title>
-      <para> Try to write some rules yourself. Read the description of the available language
-        constructs, e.g., conditions and actions and use the explanation component in order to take
-        a closer look at the rule inference. Then finally, read the rest of this document.
-      </para>
-    </section>
+    <programlisting><![CDATA[W W;]]></programlisting>
+    
+    <programlisting><![CDATA[Document{-> RETAINTYPE(SPACE)};
+Sentence;
+Document{-> FILTERTYPE(CW)};
+Sentence;
+Document{-> RETAINTYPE, FILTERTYPE};]]></programlisting>
+
   </section>
+  
   <section id="ugr.tools.tm.ae">
     <title>UIMA Analysis Engines</title>
     <para>This section gives an overview of the UIMA Analysis Engines shipped with TextMarker. The most