You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Greg Holmberg <ho...@comcast.net> on 2011/03/02 02:00:28 UTC
How to create and use a repository for UIMA annotators?
Hi UIMA users--
Recently (issue UIMA-1969) it was discovered that data resource URLs could be HTTP:
I was wondering if the same would work for component descriptor files, especially the import statements in aggregate analysis engine descriptors. For example, would this work?
<delegateAnalysisEngine key="MyAE">
<import location="http://host/com/foo/myproj/desc/MyAE.xml" />
</delegateAnalysisEngine>
Or, how about this?
<delegateAnalysisEngine key="MyAE">
<import name="myproj.desc.MyAE" />
</delegateAnalysisEngine>
and set datapath to "http://host/com/foo"
The documentation (references.pdf, section 2.2 Imports) only describes FILE:, but if it's implemented using the Java URL class, then HTTP: should work also, right?
Of course, if it can now find MyAE.xml, that doesn't mean it can find MyAE.jar.
So, how can I make UIMA find everything related to MyAE from an HTTP server? Use PEAR files? RMI codebase?
http://download.oracle.com/javase/6/docs/technotes/guides/rmi/codebase.html
My goal would be to have a place (an HTTP server) where annotator developers deploy annotators, and then analysis engine developers can just reference them from there, without having to manually install them to local disk. And, since the URL is the same from everywhere, the aggregate analysis engine description could be executed on any machine in the network without modification. I.e., the descriptor is portable.
It's sort of a "maven-like" model (i.e. when using a Nexus server). Or maybe I should just actually use maven and nexus?
Has anyone out there tried to create a "UIMA Repository" that can be directly referenced from a component descriptor file? How did you make it work?
Thanks,
Greg Holmberg
Re: How to create and use a repository for UIMA annotators?
Posted by Richard Eckart de Castilho <ec...@tk.informatik.tu-darmstadt.de>.
Hi Chris,
> I find the idea of enabling maven by packaging components in jars very
> compelling. Have you dealt with third-party code that expects to find
> resources from file system locations rather than classpath names?
I am not completely sure what you mean, so I hope my answer will satisfy you.
If you want to build a pipeline for a third-party component, that cannot deal with URLs, DKPro Core includes ResourceUtils.getUrlAsFile() [1] which bridges that for you and even supports caching. For resolving "classpath:" URLs, you can either use ResourceUtils.resolveLocation().
AnalysisEngine ae = createPrimitive(ThirdPartyAE.class,
ThirdPartyAE.PARAM_RESOURCE_FILE, getUrlAsFile(resolveLocation("classpath:/my/packaged/resource.bin"), true).getAbsolutePath());
If the third-party component supports the UIMA ResourceLoader, you should be able to configure that to resolve resources from the file-system.
Some of the components we have implemented support loading resources from the classpath. This means we can package resources like tagging models as JARs and add them as Maven dependencies as well.
DKPro includes Ant scripts that automatically create such JARs for TreeTagger models and binaries as well as for models of the Stanford Parser and NER. The generated JARs can be uploaded to a Maven repository and added to a project just like that (due to license restrictions, not to a public repository). The TreeTagger component is intelligent enough to load the correct model just by looking at the document language set in the CAS. The Stanford Parser and NER components currently can't do that, here you'd have to specify a model URL like "classpath:/resource/Classifiers/FaruquiPado/hgc_GERMAN_175M.ser.gz" (cf. [2]).
DKPro also includes a powerful base class for CollectionReaders that uses the Spring PathMatchingResourcePatternResolver [3], which is also used by uimaFIT for automatic type detection. ResourceCollectionReaderBase [4] allows you to easily create CollectionReaders capable of loading data from the file system or the classpath (or any other location/URL supported by the Spring Resource framework) using Ant-like inclusion/exclusion patterns. For example our TextReader uses that:
CollectionReader reader = createCollectionReader(TextReader.class,
TextReader.PARAM_PATH, "classpath:/data",
TextReader.PARAM_PATTERNS, new String[] { "[+]text/**/*.txt", "[-]**/broken.txt" },
TextReader.PARAM_LANGUAGE, "en");
Best,
Richard
[1] http://code.google.com/p/dkpro-core-asl/source/browse/de.tudarmstadt.ukp.dkpro.core-asl/trunk/de.tudarmstadt.ukp.dkpro.core.api.resources/src/main/java/de/tudarmstadt/ukp/dkpro/core/api/resources/ResourceUtils.java
[2] http://code.google.com/p/dkpro-core-gpl/source/browse/de.tudarmstadt.ukp.dkpro.core-gpl/trunk/de.tudarmstadt.ukp.dkpro.core.stanfordnlp/src/test/java/de/tudarmstadt/ukp/dkpro/core/stanfordnlp/StanfordNamedEntityRecognizerTest.java
[3] http://static.springsource.org/spring/docs/2.5.x/api/org/springframework/core/io/support/PathMatchingResourcePatternResolver.html
[4] http://code.google.com/p/dkpro-core-asl/source/browse/de.tudarmstadt.ukp.dkpro.core-asl/trunk/de.tudarmstadt.ukp.dkpro.core.api.io/src/main/java/de/tudarmstadt/ukp/dkpro/core/api/io/ResourceCollectionReaderBase.java
--
-------------------------------------------------------------------
Richard Eckart de Castilho
Technical Lead
Ubiquitous Knowledge Processing Lab
FB 20 Computer Science Department
Technische Universität Darmstadt
Hochschulstr. 10, D-64289 Darmstadt, Germany
phone +49 (6151) 16-7477, fax -5455, room S2/02/E225
eckartde@tk.informatik.tu-darmstadt.de
www.ukp.tu-darmstadt.de
-------------------------------------------------------------------
Re: How to create and use a repository for UIMA annotators?
Posted by Chris Roeder <ch...@ucdenver.edu>.
Richard,
I find the idea of enabling maven by packaging components in jars very
compelling. Have you dealt with third-party code that expects to find
resources from file system locations rather than classpath names?
-Chris
On 3/2/11 12:49 AM, Richard Eckart de Castilho wrote:
> Hello Greg,
>
>> It's sort of a "maven-like" model (i.e. when using a Nexus server). Or maybe I should just actually use maven and nexus?
>>
>> Has anyone out there tried to create a "UIMA Repository" that can be directly referenced from a component descriptor file? How did you make it work?
>
> We consider ourselves to have a "UIMA Repository" based on Maven - cf. DKPro Core http://code.google.com/p/dkpro-core-asl/
>
> I would like to point out that we have largely abandonded static UIMA descriptors (except type descriptors).
>
> We feel very comfortable programming on the Java level, dynamically creating descriptors using uimaFIT and running our pipelines directly from within Java (no CPE GUI or such).
> For this scenario, Maven works like a charm for us. We do not even worry too much about type systems, because we have packaged their XML descriptors and JCas
> wrappers in JARs as well and can simply add them as Maven dependencies. We use uimaFIT's automatic type system detection feature to dynamically construct a
> global type system description from all type system description files that could be found in a well-defined location in the classpath (that is, in the afore
> mentioned JARs). A short example:
>
> * add dependency on de.tudarmstadt.ukp.dkpro.core.io.text-asl (for TextReader)
> * add dependency on de.tudarmstadt.ukp.dkpro.core.tokit-asl (for BreakIteratorSegmenter)
> * add dependency on de.tudarmstadt.ukp.dkpro.core.dictionaryannotator-asl (for DictionaryAnnotator)
> * dependency on uimaFIT automatically added (for CASDumpWriter)
> * dependencies on type systems and JCas wrappers automatically added by Maven
>
> Then we can immediately assemble and run a pipeline:
>
> CollectionReader reader = createCollectionReader(TextReader.class,
> TextReader.PARAM_PATH, "src/test/resources/text",
> TextReader.PARAM_PATTERNS, new String[] { "[+]*.txt", "[-]broken.txt" },
> TextReader.PARAM_LANGUAGE, "en");
>
> AnalysisEngine tokenizer = createPrimitive(BreakIteratorSegmenter.class);
>
> AnalysisEngine nameFinder = createPrimitive(DictionaryAnnotator.class,
> DictionaryAnnotator.PARAM_PHRASE_FILE, "src/test/resources/dictionaries/names.txt",
> DictionaryAnnotator.PARAM_ANNOTATION_TYPE, Name.class.getName());
>
> AnalysisEngine writer = createPrimitive(CASDumpWriter.class,
> CASDumpWriter.PARAM_OUTPUT_FILE, "target/output.txt");
>
> SimplePipeline.runPipeline(reader, tokenizer, nameFinder, writer);
>
> Notice that no line references a type system whatsoever. This is because we let uimaFIT automatically scan the classpath and simply make all
> types it finds available to every created component.
>
> Our approach seems to work great for our researchers to assemble and run pipelines on a single machine. We do currently not scale out UIMA.
>
> Cheers,
>
> Richard
>
--
Christophe (Chris) Roeder
Software Developer, Professional Research Assistant
Center for Computational Pharmacology, University of Colorado Denver
12801 E 17th Ave, MS 8303, Aurora, CO 80045 USA
chris.roeder@ucdenver.edu / tel: (303) 724-7574
Re: How to create and use a repository for UIMA annotators?
Posted by Richard Eckart de Castilho <ri...@gmail.com>.
Am 07.05.2013 um 03:47 schrieb swirl <sw...@yahoo.com>:
> Hi Richard,
> Would you mind showing me how the uimafit is able to "dynamically construct
> a global type system description from all type system description files that
> could be found in a well-defined location in the classpath".
>
> Do you rely on using the umiafit/types.txt file?
> If so, how you specify it such that it is able to pick up the type
> description files in the classpath? I looked into the
> de.tudarmstadt.ukp.dkpro.core.tokit-asl-1.4.0.jar but there are no type
> description file inside the JAR itself.
Check out
https://code.google.com/p/uimafit/wiki/TypeDescriptorDetection
Cheers,
-- Richard
AW: How to create and use a repository for UIMA annotators?
Posted by Ar...@bka.bund.de.
Hi,
In my opinion, the best way to do it, is to use an empty type system with the collection reader. You can create one with TypeSystemDescriptionFactory.createTypeSystemDescription(). If one of your annotators needs a type system, add it there, e. g. AnalysisEngineFactory.createPrimitiveDescription(Annotator.class, typeSystem, ...). As one of the last component of your pipeline use a consumer, which writes the overall type system description to file. The core of the consumer is
public final void process(final CAS cas) throws AnalysisEngineProcessException {
try {
TypeSystemUtil.typeSystem2TypeSystemDescription(cas.getTypeSystem()).toXML(new FileOutputStream(new File(mOutputFilePathString)));
} catch ...
}
}
Cheers,
Armin
-----Ursprüngliche Nachricht-----
Von: swirl [mailto:swirlobt@yahoo.com]
Gesendet: Dienstag, 7. Mai 2013 03:47
An: user@uima.apache.org
Betreff: Re: How to create and use a repository for UIMA annotators?
Richard Eckart de Castilho <ec...@...> writes:
>
> Hello Greg,
>
> > It's sort of a "maven-like" model (i.e. when using a Nexus server).
> > Or
maybe I should just actually use
> maven and nexus?
> >
> > Has anyone out there tried to create a "UIMA Repository" that can be
directly referenced from a component
> descriptor file? How did you make it work?
>
> We consider ourselves to have a "UIMA Repository" based on Maven - cf.
DKPro Core http://code.google.com/p/dkpro-core-asl/
>
> I would like to point out that we have largely abandonded static UIMA
descriptors (except type descriptors).
>
> We feel very comfortable programming on the Java level, dynamically
creating descriptors using uimaFIT
> and running our pipelines directly from within Java (no CPE GUI or such).
> For this scenario, Maven works like a charm for us. We do not even
> worry
too much about type systems, because
> we have packaged their XML descriptors and JCas wrappers in JARs as
> well and can simply add them as Maven dependencies. We
use uimaFIT's automatic type
> system detection feature to dynamically construct a global type system
> description from all type system description files that
could be found in a
> well-defined location in the classpath (that is, in the afore
> mentioned JARs). A short example:
>
> * add dependency on de.tudarmstadt.ukp.dkpro.core.io.text-asl (for
TextReader)
> * add dependency on de.tudarmstadt.ukp.dkpro.core.tokit-asl (for
BreakIteratorSegmenter)
> * add dependency on
> de.tudarmstadt.ukp.dkpro.core.dictionaryannotator-
asl (for DictionaryAnnotator)
> * dependency on uimaFIT automatically added (for CASDumpWriter)
> * dependencies on type systems and JCas wrappers automatically added
> by
Maven
>
> Then we can immediately assemble and run a pipeline:
>
> CollectionReader reader = createCollectionReader(TextReader.class,
> TextReader.PARAM_PATH, "src/test/resources/text",
> TextReader.PARAM_PATTERNS, new String[] { "[+]*.txt", "[-
]broken.txt" },
> TextReader.PARAM_LANGUAGE, "en");
>
> AnalysisEngine tokenizer =
createPrimitive(BreakIteratorSegmenter.class);
>
> AnalysisEngine nameFinder = createPrimitive(DictionaryAnnotator.class,
> DictionaryAnnotator.PARAM_PHRASE_FILE,
"src/test/resources/dictionaries/names.txt",
> DictionaryAnnotator.PARAM_ANNOTATION_TYPE,
> Name.class.getName());
>
> AnalysisEngine writer = createPrimitive(CASDumpWriter.class,
> CASDumpWriter.PARAM_OUTPUT_FILE, "target/output.txt");
>
> SimplePipeline.runPipeline(reader, tokenizer, nameFinder, writer);
>
> Notice that no line references a type system whatsoever. This is
> because
we let uimaFIT automatically scan
> the classpath and simply make all
> types it finds available to every created component.
>
> Our approach seems to work great for our researchers to assemble and
> run
pipelines on a single machine. We do
> currently not scale out UIMA.
>
> Cheers,
>
> Richard
>
Hi Richard,
Would you mind showing me how the uimafit is able to "dynamically construct a global type system description from all type system description files that could be found in a well-defined location in the classpath".
Do you rely on using the umiafit/types.txt file?
If so, how you specify it such that it is able to pick up the type description files in the classpath? I looked into the de.tudarmstadt.ukp.dkpro.core.tokit-asl-1.4.0.jar but there are no type description file inside the JAR itself.
Re: How to create and use a repository for UIMA annotators?
Posted by swirl <sw...@yahoo.com>.
Richard Eckart de Castilho <ec...@...> writes:
>
> Hello Greg,
>
> > It's sort of a "maven-like" model (i.e. when using a Nexus server). Or
maybe I should just actually use
> maven and nexus?
> >
> > Has anyone out there tried to create a "UIMA Repository" that can be
directly referenced from a component
> descriptor file? How did you make it work?
>
> We consider ourselves to have a "UIMA Repository" based on Maven - cf.
DKPro Core http://code.google.com/p/dkpro-core-asl/
>
> I would like to point out that we have largely abandonded static UIMA
descriptors (except type descriptors).
>
> We feel very comfortable programming on the Java level, dynamically
creating descriptors using uimaFIT
> and running our pipelines directly from within Java (no CPE GUI or such).
> For this scenario, Maven works like a charm for us. We do not even worry
too much about type systems, because
> we have packaged their XML descriptors and JCas
> wrappers in JARs as well and can simply add them as Maven dependencies. We
use uimaFIT's automatic type
> system detection feature to dynamically construct a
> global type system description from all type system description files that
could be found in a
> well-defined location in the classpath (that is, in the afore
> mentioned JARs). A short example:
>
> * add dependency on de.tudarmstadt.ukp.dkpro.core.io.text-asl (for
TextReader)
> * add dependency on de.tudarmstadt.ukp.dkpro.core.tokit-asl (for
BreakIteratorSegmenter)
> * add dependency on de.tudarmstadt.ukp.dkpro.core.dictionaryannotator-
asl (for DictionaryAnnotator)
> * dependency on uimaFIT automatically added (for CASDumpWriter)
> * dependencies on type systems and JCas wrappers automatically added by
Maven
>
> Then we can immediately assemble and run a pipeline:
>
> CollectionReader reader = createCollectionReader(TextReader.class,
> TextReader.PARAM_PATH, "src/test/resources/text",
> TextReader.PARAM_PATTERNS, new String[] { "[+]*.txt", "[-
]broken.txt" },
> TextReader.PARAM_LANGUAGE, "en");
>
> AnalysisEngine tokenizer =
createPrimitive(BreakIteratorSegmenter.class);
>
> AnalysisEngine nameFinder = createPrimitive(DictionaryAnnotator.class,
> DictionaryAnnotator.PARAM_PHRASE_FILE,
"src/test/resources/dictionaries/names.txt",
> DictionaryAnnotator.PARAM_ANNOTATION_TYPE, Name.class.getName());
>
> AnalysisEngine writer = createPrimitive(CASDumpWriter.class,
> CASDumpWriter.PARAM_OUTPUT_FILE, "target/output.txt");
>
> SimplePipeline.runPipeline(reader, tokenizer, nameFinder, writer);
>
> Notice that no line references a type system whatsoever. This is because
we let uimaFIT automatically scan
> the classpath and simply make all
> types it finds available to every created component.
>
> Our approach seems to work great for our researchers to assemble and run
pipelines on a single machine. We do
> currently not scale out UIMA.
>
> Cheers,
>
> Richard
>
Hi Richard,
Would you mind showing me how the uimafit is able to "dynamically construct
a global type system description from all type system description files that
could be found in a well-defined location in the classpath".
Do you rely on using the umiafit/types.txt file?
If so, how you specify it such that it is able to pick up the type
description files in the classpath? I looked into the
de.tudarmstadt.ukp.dkpro.core.tokit-asl-1.4.0.jar but there are no type
description file inside the JAR itself.
Re: How to create and use a repository for UIMA annotators?
Posted by Richard Eckart de Castilho <ec...@tk.informatik.tu-darmstadt.de>.
Hello Greg,
> It's sort of a "maven-like" model (i.e. when using a Nexus server). Or maybe I should just actually use maven and nexus?
>
> Has anyone out there tried to create a "UIMA Repository" that can be directly referenced from a component descriptor file? How did you make it work?
We consider ourselves to have a "UIMA Repository" based on Maven - cf. DKPro Core http://code.google.com/p/dkpro-core-asl/
I would like to point out that we have largely abandonded static UIMA descriptors (except type descriptors).
We feel very comfortable programming on the Java level, dynamically creating descriptors using uimaFIT and running our pipelines directly from within Java (no CPE GUI or such).
For this scenario, Maven works like a charm for us. We do not even worry too much about type systems, because we have packaged their XML descriptors and JCas
wrappers in JARs as well and can simply add them as Maven dependencies. We use uimaFIT's automatic type system detection feature to dynamically construct a
global type system description from all type system description files that could be found in a well-defined location in the classpath (that is, in the afore
mentioned JARs). A short example:
* add dependency on de.tudarmstadt.ukp.dkpro.core.io.text-asl (for TextReader)
* add dependency on de.tudarmstadt.ukp.dkpro.core.tokit-asl (for BreakIteratorSegmenter)
* add dependency on de.tudarmstadt.ukp.dkpro.core.dictionaryannotator-asl (for DictionaryAnnotator)
* dependency on uimaFIT automatically added (for CASDumpWriter)
* dependencies on type systems and JCas wrappers automatically added by Maven
Then we can immediately assemble and run a pipeline:
CollectionReader reader = createCollectionReader(TextReader.class,
TextReader.PARAM_PATH, "src/test/resources/text",
TextReader.PARAM_PATTERNS, new String[] { "[+]*.txt", "[-]broken.txt" },
TextReader.PARAM_LANGUAGE, "en");
AnalysisEngine tokenizer = createPrimitive(BreakIteratorSegmenter.class);
AnalysisEngine nameFinder = createPrimitive(DictionaryAnnotator.class,
DictionaryAnnotator.PARAM_PHRASE_FILE, "src/test/resources/dictionaries/names.txt",
DictionaryAnnotator.PARAM_ANNOTATION_TYPE, Name.class.getName());
AnalysisEngine writer = createPrimitive(CASDumpWriter.class,
CASDumpWriter.PARAM_OUTPUT_FILE, "target/output.txt");
SimplePipeline.runPipeline(reader, tokenizer, nameFinder, writer);
Notice that no line references a type system whatsoever. This is because we let uimaFIT automatically scan the classpath and simply make all
types it finds available to every created component.
Our approach seems to work great for our researchers to assemble and run pipelines on a single machine. We do currently not scale out UIMA.
Cheers,
Richard
--
-------------------------------------------------------------------
Richard Eckart de Castilho
Technical Lead
Ubiquitous Knowledge Processing Lab
FB 20 Computer Science Department
Technische Universität Darmstadt
Hochschulstr. 10, D-64289 Darmstadt, Germany
phone +49 (6151) 16-7477, fax -5455, room S2/02/E225
eckartde@tk.informatik.tu-darmstadt.de
www.ukp.tu-darmstadt.de
-------------------------------------------------------------------