You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@uima.apache.org by sc...@apache.org on 2012/12/11 23:32:46 UTC

svn commit: r1420472 [2/2] - in /uima/sandbox/trunk/uimaFIT/uimafit-parent: ./ .settings/ scripts/ wiki/

Added: uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/TwoSidesOfUimaFIT.wiki
URL: http://svn.apache.org/viewvc/uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/TwoSidesOfUimaFIT.wiki?rev=1420472&view=auto
==============================================================================
--- uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/TwoSidesOfUimaFIT.wiki (added)
+++ uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/TwoSidesOfUimaFIT.wiki Tue Dec 11 22:32:42 2012
@@ -0,0 +1,60 @@
+= Introduction =
+
+While uimaFIT provides many features for a UIMA developer, there are two overarching themes that most features fall under.  These two sides of uimaFIT are, while complementary, largely independent of each other.  One of the beauties of uimaFIT is that a developer that uses one side of uimaFIT extensively is not required to use the other side at all.  
+
+= Simplify Component Implementation = 
+
+The first broad theme of uimaFIT provides features that *simplify component implementation*.  Our favorite example of this is the @!ConfigurationParameter annotation which allows you to annotate a member variable as a configuration parameter.  This annotation in combination with the method !ConfigurationParameterInitializer.initialize completely automates the process of initializing member variables with values from the !UimaContext passed into your analysis engine's initialize method.  Similarly, the annotation @!ExternalResource annotation in combination with the method !ExternalResourceInitializer.initialize completely automates the binding of an external resource as defined in the !UimaContext to a member variable.  Dispensing with manually writing the code that performs these two tasks reduces effort, eliminates verbose and potentially buggy boiler-plate code, and makes implementing a UIMA component more enjoyable.  Consider, for example, a member variable that is of ty
 pe 'Locale'.  With uimaFIT you can simply annotate the member variable with '@!ConfigurationParameter' and have your initialize method automatically initialize the variable correctly with a string value in the !UimaContext such as "en_US".  
+
+= Simplify Component Instantiation = 
+
+The second broad theme of uimaFIT provides features that *simplify component instantiation*.  Working with UIMA, have you ever said to yourself "but I just want to tag some text!?"  What does it take to "just tag some text?"  Here's a list of things you must do with the traditional approach:
+
+ * wrap your tagger as a UIMA analysis engine
+ * write a descriptor file for your analysis engine
+ * write a CAS consumer that produces the desired output
+ * write another descriptor file for the CAS consumer
+ * write a descriptor file for a collection reader
+ * write a descriptor file that describes a pipeline
+ * invoke the Collection Processing Manager with your pipeline descriptor file
+
+Each of these steps has its own pitfalls and can be rather time consuming.  This is a rather unsatisfying answer to our simple desire to just tag some text.  With uimaFIT you can literally eliminate all of these steps.  Here's a simple snippet of Java code that demonstrates "tagging some text" with uimaFIT:
+
+{{{
+
+TypeSystemDescription typeSystemDescription = createTypeSystemDescription();
+
+JCas jCas = createJCas(typeSystemDescription);
+
+jCas.setDocumentText("some text");
+
+AnalysisEngine tokenizer = createPrimitive(MyTokenizer.class, typeSystemDescription);
+
+AnalysisEngine tagger = createPrimitive(MyTagger.class, typeSystemDescription);
+
+runPipeline(jCas, tokenizer, tagger);
+
+for(Token token : iterate(jCas, Token.class)){
+    System.out.println(token.getTag());
+}
+
+}}}
+
+This code assumes several static method imports (e.g. createTypeSystemDescription) provided by uimaFIT for brevity.  And while the terseness of this code won't make a python programmer blush - it is certainly much easier than the seven steps outlined above!  
+
+uimaFIT provides mechanisms to instantiate and run UIMA components programmatically with or without descriptor files.  For example, if you have a descriptor file for your analysis engine defined by !MyTagger.java (as shown above), then you can instead instantiate the analysis engine with:
+
+{{{
+AnalysisEngine tagger = createAnalysisEngine("mypackage.MyTagger");
+}}}
+
+This will find the descriptor file 'mypackage/!MyTagger.xml' by name.  Similarly, you can find a descriptor file by location with createAnalysisEngineFromPath().  However, if you want to dispense with XML descriptor files altogether (and you probably do), you can use the method createPrimitive() as shown above.  One of the driving motivations for creating the second side of uimaFIT is our frustration with descriptor files and our desire to eliminate them.  Descriptor files are difficult to maintain because they are generally tightly coupled with java code, they decay without warning, they are wearisome to test, and they proliferate, among other reasons.  
+
+= Is this cheating? =
+
+One question that is often raised by new uimaFIT users is whether or not it breaks the "UIMA way".  That is, does adopting uimaFIT lead me down a path of creating UIMA components and systems that are incompatible with the traditional UIMA approach?  The answer to this question is no.  For starters, uimaFIT does not skirt the UIMA mechanism of describing components - it only skips the XML part of it.  For example, when the method createPrimitive is called (as shown above) an !AnalysisEngineDescription is created for the !AnalysisEngine.  This is the same object type that is instantiated when a descriptor file is used.  So, instead of parsing XML to instantiate an !AnalysisEngineDescription, uimaFIT uses a factory method to instantiate it from method parameters.  One of the happy benefits of this approach is that for a given !AnalysisEnginedDescription (which can be obtained directly with createPrimitiveDescription) you can generate an XML descriptor file using !AnalysisEngine
 Description.toXML().  So, uimaFIT actually provides a very simple and direct path for *generating* XML descriptor files rather than manually creating and maintaining them!  
+
+It is also useful to clarify that if you only want to use one side or the other of uimaFIT, then you are free to do so.  This is possible precisely because uimaFIT does not workaround UIMA's mechanisms for describing components but rather uses them directly.  For example, if the only thing you want to use in uimaFIT is the '@!ConfigurationParameter', then you can do so without worrying about what effect this will have on your descriptor files.  This is because your analysis engine will be initialized with exactly the same !UimaContext regardless of whether you instantiate your analysis engine in the "UIMA way" or use one of uimaFIT's factory methods.  Similarly, a UIMA component does not need to be annotated with @!ConfiguratioParameter for you to make use of the createPrimitive method.  This is because when you pass configuration parameter values in to the createPrimitive method, they are added to an !AnalysisEngineDescription which is used by UIMA to populate a !UimaContex
 t - just as it would if you used a descriptor file.  
+
+= Conclusion =
+Because uimaFIT can be used to simplify component implementation and instantiation it is easy to assume that you can't do one without the other.  This page has demonstrated that while these two sides of uimaFIT complement each other, they are not coupled together and each can be effectively used without the other.  Similarly, by understanding how uimaFIT uses the UIMA component description mechanisms directly, one can be assured that uimaFIT enables UIMA development that is compatible and consistent with the UIMA standard and APIs.  
\ No newline at end of file

Added: uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/TypeDescriptorDetection.wiki
URL: http://svn.apache.org/viewvc/uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/TypeDescriptorDetection.wiki?rev=1420472&view=auto
==============================================================================
--- uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/TypeDescriptorDetection.wiki (added)
+++ uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/TypeDescriptorDetection.wiki Tue Dec 11 22:32:42 2012
@@ -0,0 +1,106 @@
+<h1>External Resources</h1>
+
+<wiki:toc max_depth="2" />
+
+UIMA requires that types that are used in the CAS are defined in XML files - so-called type system descriptions (TSD). Whenever a UIMA component is created, it must be associated with such a type system. While it is possible to manually load the type system descriptors and pass them to each
+UIMA component and to each created CAS, it is quite inconvenient to do so. For this reason, uimaFIT supports the automatic detection of such files in the classpath. Thus is becomes possible for a UIMA component provider to have component's type automatically detected and thus the components becomes immediately usable by adding it to the classpath.
+
+= Making types auto-detectable =
+
+The provider of a type system should create a file `META-INF/org.uimafit/types.txt` in the classpath. This file should define the locations of the type system descriptions. Assume that a type `org.uimafit.type.Token` is specified  in the TSD `org/uimafit/type/Token.xml`, then the file should have the following contents:
+
+{{{
+classpath*:org/uimafit/type/Token.xml
+}}}
+
+To specify multiple TSDs, add additonal lines to the file. If you have a large number of TSDs, you may prefer to add a pattern. Assume that we have a large number of TSDs under `org/uimafit/type`, we can use the following pattern which recursively scans the package `org.uimafit.type` and all sub-packages for XML files and tries to load them as TSDs.
+
+{{{
+classpath*:org/uimafit/type/**/*.xml
+}}}
+
+Try to design your packages structure in a way that TSDs and JCas wrapper classes generated from them are separate from the rest of your code. 
+
+If it is not possible or inconvenient to add the `types.txt` file, patterns can also be specified using a system property. Multiple patterns may be specified separated by semicolon:
+
+{{{
+org.uimafit.type.import_pattern=classpath*:org/uimafit/type/**/*.xml
+}}}
+
+= Using type auto-detection =
+
+The auto-detected type system can be obtained from the `TypeSystemDescriptionFactory`:
+
+{{{
+TypeSystemDescription tsd = TypeSystemDescriptionFactory.createTypeSystemDescription()
+}}}
+
+Popular factory methods also support auto-detection:
+
+{{{
+AnalysisEngine ae = createPrimitive(MyEngine.class);
+}}}
+
+= Multiple `META-INF/org.uimafit/types.txt` files =
+
+uimaFIT supports multiple `types.txt` files in the classpath (e.g. in differnt JARs). 
+
+The `types.txt` files are located via Spring using the classpath search pattern: 
+
+{{{
+TYPE_MANIFEST_PATTERN = "classpath*:META-INF/org.uimafit/types.txt" 
+}}}
+
+This resolves to a list URLs pointing to ALL `types.txt`files. The resolved URLs are unique and will point either to a specific point in the file system or into a specific JAR. These URLs can be handled by the standard Java URL loading mechanism. 
+
+Example: 
+
+{{{
+jar:/path/to/syntax-types.jar!/META-INF/org.uimafit/types.txt 
+jar:/path/to/token-types.jar!/META-INF/org.uimafit/types.txt
+}}}
+ 
+uimaFIT then reads all patters from all of these URLs and uses these to search the classpath again. The patterns now resolve to a list of URLs pointing to the individual type system XML descriptors. All of these URLs are collected in a set to avoid duplicate loading (for performance optimization - not strictly necessary because the UIMA type system merger can handle compatible duplicates). Then the descriptors are loaded into memory and merged using the standard UIMA type system merger (`CasCreationUtils.mergeTypeSystems()`). 
+
+Example: 
+
+{{{
+jar:/path/to/syntax-types.jar!/desc/types/Syntax.xml 
+jar:/path/to/token-types.jar!/org/foobar/typesystems/Tokens.xml 
+}}}
+Voilá, the result is a type system covering all types could be found in the classpath. 
+
+I recommend 
+  # to put type system descriptors into packages resembling a namespace you "own" and to use a package-scoped wildcard search 
+{{{ 
+classpath*:org/uimafit/type/**/*.xml`
+}}}
+  # or when putting descriptors into a "well-known" package like `desc.type`, that `types.txt` file  should explicitly list all type system descriptors instead of using a wildcard search or 
+{{{
+classpath*:desc/type/Token.xml 
+classpath*:desc/type/Syntax.xml 
+}}}
+
+Method 1 should be preferred. Both methods can be mixed. 
+
+= Performance note and caching =
+
+Currently uimaFIT evaluates the patterns for TSDs once and caches the locations, but not the actual merged type system description. A rescan can be forced using `TypeSystemDescriptionFactory.forceTypeDescriptorsScan()`. This may change in future.
+
+= Potential problems =
+
+The mechanism works fine. However, there are specific issues with Java in general that one should be aware of.
+
+== m2eclipse fails to copy descriptors to target/classes ==
+
+There seems to be a bug in m2eclipse that causes resources not always to be copied to `target/classes`. If UIMA complains about type definitions missing at runtime, try to clean/rebuild your project and carefully check the m2eclipse console in the console view for error messages that might cause m2eclipse to abort.
+
+== Class version conflicts ==
+
+A problem can occur if you end up having multiple incompatible versions of the same type system in the classpath. This is a general problem and not related to the auto-detection feature. It is the same as when you have incompatible version of a particular class (e.g. JCas wrapper or some third-party-library) in the classpath. The behavior of the Java Classloader is undefined in that case. The detection will do its best to try and load everything it can find, but the UIMA type system merger may barf or you may end up with undefined behavior at runtime because one of the class versions is used at random. 
+
+== Classes and resources in the default package ==
+
+It is bad practice to place classes into the default (unnamed) package. In fact it is not possible to import classes from the default package in another class. Similarly it is a bad idea to put resources at the root of the classpath. The Spring documentation on resources [http://static.springsource.org/spring/docs/3.0.x/reference/resources.html#resources-app-ctx-wildcards-in-resource-paths explains this in detail].
+
+For this reason the `types.txt` resides in `/META-INF/org.uimafit` and I suggest that type system descriptors reside either in a proper package like `/org/foobar/typesystems/XXX.xml` or in `/desc/types/XXX.xml`. 
\ No newline at end of file

Added: uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/UserSetup.wiki
URL: http://svn.apache.org/viewvc/uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/UserSetup.wiki?rev=1420472&view=auto
==============================================================================
--- uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/UserSetup.wiki (added)
+++ uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/UserSetup.wiki Tue Dec 11 22:32:42 2012
@@ -0,0 +1,19 @@
+The following instructions describe how to add uimaFIT to your project's classpath.  
+
+= Maven Users = 
+
+If you use Maven, then uimaFIT can be added to your project by simply adding uimaFIT as a project dependency by adding the following snippet of XML to your pom.xml file:
+
+{{{
+<dependency>
+  <groupId>org.uimafit</groupId>
+  <artifactId>uimafit</artifactId>
+  <version>(version-number-here)</version>
+</dependency>
+}}}
+
+Please see the project [http://uimafit.googlecode.com front page] for the current version number.  uimaFIT distributions are hosted by Maven Central and so no repository needs to be added to your pom.xml file.  
+
+= Non-maven Users = 
+
+If you do not build with Maven, then download uimaFIT from the [http://code.google.com/p/uimafit/downloads/list downloads page].  The file name should be uimafit-(version)-plus-dependencies.zip.  Download and unpack this file.  The contents of the resulting upacked directory will contain a directory called "lib".  Add all of the files in this directory to your classpath.
\ No newline at end of file

Added: uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/WorkingWithSofaViews.wiki
URL: http://svn.apache.org/viewvc/uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/WorkingWithSofaViews.wiki?rev=1420472&view=auto
==============================================================================
--- uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/WorkingWithSofaViews.wiki (added)
+++ uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/WorkingWithSofaViews.wiki Tue Dec 11 22:32:42 2012
@@ -0,0 +1,53 @@
+#summary Describes nuances of Sofas, CASes and CAS Views (in development)
+
+= Introduction =
+
+There are some use-cases for copying text and annotations between CASes, and views. There are various Annotators and classes that help deal with these situations available from the different projects:
+
+ * UIMA has the CasCopier that copies annotations between separate CASes, preserving view names.
+ * uimaFit has the ViewTextCopierAnnotator that copies text between views in the same CAS.
+ * Phillip Wetzler has progress on code that copies annotations documented in an issue at uimaFIT (46), and ClearTK (175). The code is attached to the uimaFIT issue. It needs test code.
+
+This page describes the problems and the tools. A glossary is included. 
+
+
+= Details =
+
+==CASes, Views and Multi-Cas==
+
+In a simple UIMA annotator, a CAS is a structure that contains FeatureSets and the text or the Sofa, the subject of Annotation. 
+
+A more involved annotator can work with a CAS that has more than one view on the CAS. Such an annotator uses a Multi-View CAS and is said to be sofa aware.  A Multi-View CAS is a base CAS that contains views, but has no Sofa or indeces (or FeatureSets). Each view has a sofa, indeces and FeatureSets. The name sofa-aware comes from a variable in PrimitiveAnalysisEngine_impl (mSofaAware). An annotator written for single view CASes can be passed a Multi-CAS: it will always access the _InitialView because the surrounding AnalysisComponent picks it out and sends it to the AnalysisEngine. An annotator written for multi-view CAS, when passed a single-view CAS will only have the _InitialView available.
+
+An annotator's process(Jcas) method is applicable to both types of CASes (single and multi-view).
+
+An annotator is sofa-aware if it has been annotated as such with sofaCapabilities (see 6.2.3 of UTDG) . A sofa-aware annotator must get a view to work on since the top-level CAS has no sofa or featureSets. This is accomplished with jcas.getView("_InitialView"). A sofa-unaware annotator is always passed the _InitialView, the default.
+
+Any annotator, sofa-aware or sofa-unaware, can say jcas.getView("_InitialView") whether sofa-aware or not.
+
+Don't be lulled into a false sense of control when writing pipelines in uimaFIT. Just because you can pass a view to the process method of an annotator, does not mean that view will be available inside the process method. When anntoation engines are created in uimaFIT, like in UIMA, it is actually an analysis component that surrounds the analysis engine that is created. The process method of the component works to pass the right view to the analysis engine in case it is a cas-unaware or single-view component. The name sofa-aware comes from a a member variable, mSofaAware, in PrimitiveAnalysisEngine_impl. (personal experience)
+
+When copying annotations from one view to another, using clone() for eample, make sure the sofa feature set is changed to indicate the new view. (need details on exactly how to do this)
+
+
+==Related Tools==
+
+ * *CasCopier* use this to copy between distinct CASes
+ * *....ViewCreator* use this to safely create views
+ * *ViewTextCopierAnnotator* use this to copy text between views
+ * *foo* use this to copy annotations between views. (untested) 
+
+----
+=Glossary=
+ * *CAS* Common Annotation/Analysis Structure, an array of FeatureSets, various indeces, and the DocumentText (Ch. 1 UTDG) 
+ * *base CAS* a base CAS is the top-level CAS that contains views. It is different from a simple CAS that contains no views in that the base-CAS has no sofa or indexes.
+ * *JCas* A Java interface to a CAS (Ch. 1 UTDG)
+ * *FeatureStructure* Cas Feature Structure type, types of annotations described in a TypeSystem.
+ * *Feature*, attributes of a FeatureStructure (?)
+ * *Sofa Subject of Annotation*  The Sofa is the text under consideration. The view is the Sofa and related annotations. They are often used interchangeably.   
+ * *multi-View*, *multiple CAS views* UIMA supports a kind of CAS that hs more than one view. Each view has its own Sofa and related FeatureStructures. (Ch. 6 "Mutliple CAS Views of an Artifact" in the UTDG)
+ * *sofa-aware* A sofa-aware annotator can work with different views.  A non sofa-aware, or sofa-unaware, annotator will always work on the default view, _InitialView. The name comes from a member variable in ______ in UIMA.
+ * *subview*
+ 
+
+UTDG: UIMA Tutorial and Developer's Guide
\ No newline at end of file

Added: uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/WritingTests.wiki
URL: http://svn.apache.org/viewvc/uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/WritingTests.wiki?rev=1420472&view=auto
==============================================================================
--- uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/WritingTests.wiki (added)
+++ uima/sandbox/trunk/uimaFIT/uimafit-parent/wiki/WritingTests.wiki Tue Dec 11 22:32:42 2012
@@ -0,0 +1,26 @@
+#summary helpful suggestions for writing tests
+
+*under construction*
+
+= Introduction =
+
+Writing tests without uimaFIT can be a laborious process that results in fragile tests that are ver verbose and break easily when code is refactored.  This page demonstrates how you can write tests that are both concise and robust.  Here's an outline of how you might create a test for a UIMA component _*without*_ uimaFIT:
+
+ # write a descriptor file that configures your component appropriately for the test.  This requires a minimum of 30-50 lines of XML.
+ # begin a test with 5-10 lines of code that instantiate the e.g. analysis engine.
+ # run the analysis engine against some text and test the contents of the CAS.
+ # repeat steps 1-3 for your next test usually by copying the descriptor file, renaming it, and changing e.g. configuration parameters.
+
+If you have gone through the pain of creating tests like these and then decided you should refactor your code, then you know how tedious it is to maintain them.  
+
+Instead of pasting variants of the setup code (see step 2) into other tests we began to create a library of utility methods that we could call which helped shorten our code.  We extended these methods so that we could instantiate our components directly without a descriptor file.  These utility methods became the initial core of uimaFIT.  
+
+= Examples =
+
+ * There are a number of examples of unit tests in both the test suite for the uimafit project and the uimafit-examples project.  In particular, there are some well-documented unit tests in the latter which can be found in [http://code.google.com/p/uimafit/source/browse/trunk/uimaFIT-examples/src/test/java/org/uimafit/examples/tutorial/ex1/RoomNumberAnnotator1Test.java RoomNumberAnnotator1Test.java]
+ * You can improve your testing strategy by introducing a !TestBase class such as the one found in [http://code.google.com/p/uimafit/source/browse/trunk/uimaFIT-examples/src/test/java/org/uimafit/examples/tutorial/ExamplesTestBase.java ExamplesTestBase.java].  This class is intended as a super class for your other test classes and sets up a JCas that is always ready to use along with a !TypeSystemDescription and a !TypePriorities.  An example test that subclasses from !ExamplesTestBase is [http://code.google.com/p/uimafit/source/browse/trunk/uimaFIT-examples/src/test/java/org/uimafit/examples/tutorial/ex1/RoomNumberAnnotator2Test.java RoomNumberAnnotator2Test.java]
+ * Most !AnalysisEngines that you want to test will generally be downstream of many other components that add annotations to the CAS.  These annotations will likely need to be in the CAS so that a downstream !AnalysisEngine will do something sensible.  This poses a problem for tests because it may be undesirable to set up and run an entire pipeline every time you want to test a downstream !AnalysisEngine.  Furthermore, such tests can become fragile in the face of behavior changes to upstream components.  For this reason, it can be advantageous to serialize a CAS as an XMI file and use this as a starting point rather than running an entire pipeline.  An example of this approach can be found in [http://code.google.com/p/uimafit/source/browse/trunk/uimaFIT-examples/src/test/java/org/uimafit/examples/xmi/XmiTest.java XmiTest.java].  
+
+= Tips & Tricks =
+The package org.uimafit.testing provides some utility classes that can be handy when writing tests for UIMA components.  You may find the following suggestions useful:
+ * add a org.uimafit.testing.factory.!TokenBuilder instance to your !TestBase class.  An example of this can be found in [http://code.google.com/p/uimafit/source/browse/trunk/uimaFIT/src/test/java/org/uimafit/ComponentTestBase.java ComponentTestBase.java].  This makes it easy to add tokens and sentences to the CAS you are testing which is a common task for many tests.  
\ No newline at end of file