You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Alexander Klenner <al...@scai.fraunhofer.de> on 2011/09/16 10:43:59 UTC
merging CAS objects
Hello,
I have a question concerning the merging of different UIMA pipelines. Say I have 3 different annotators that work on the same document (The CAS sofa data is identical for each of the pipelines) They do this parallel and all of them produce different annotations but in a sofa with the same name(_textView). Finally I have 3 serialized XCAS files in three different folders, coming from different nodes of a cluster.
Is there an UIMA conform way to merge the corresponding xml files into one CAS object that has all the annotations of the three separate files? I could easily do this with a non uima java class that just adds all the annotation information into one file. Since the sofa data is the same, the offset information of the annotations will be correct, but I'd rather stay in the UIMA context.
Cheers,
Alex
--
Dipl. Bioinformatiker Alexander G. Klenner
Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI)
Schloss Birlinghoven, D-53754 Sankt Augustin
Tel.: +49 - 2241 - 14 - 2736
E-mail: alexander.garvin.klenner@scai.fraunhofer.de
Internet: http://www.scai.fraunhofer.de
Re: merging CAS objects
Posted by Burn Lewis <bu...@gmail.com>.
Yes, UIMA-AS will do the merging for you if you run the annotators in a
parallel step with a custom flow controller. For an example
see: examples/descriptors/flow_controller/MeetingDetectorTAE_AdvancedFixedFlow.xml
Assuming the annotators are independent of one another, the resultant CAS
would be equivalent to running them in sequence.
Throughput will also be improved if you use a multi-core machine or deploy
some or all of the annotators remotely.
A CasMultiplier could be used if you already have produced the output XMI
CASes, but would require user code to check that the sofa data is the same
and to copy the annotations into the new output CAS.
~Burn
Re: merging CAS objects
Posted by Chris Roeder <ch...@ucdenver.edu>.
The org.apache.uima.cas.impl.XmiCasDeserializer
takes an existing CAS that may contain annotations
already. I've used this when pulling annotations
in from serialized CASes in various combinations.
-Chris
On 9/16/11 3:48 AM, Jörn Kottmann wrote:
> Hello,
>
> depending on your application it might be just as fast if you just have
> one CAS and then put it
> through your three analysis pipelines, one after another. When you do it
> this way your throughput can be just
> as high, only the latency goes up.
>
> When you have multiple machines or a cluster you might want to use
> UIMA-AS. As far as I know
> you can use a parallel step in the flow, which also does the merging for
> you.
>
> Jörn
>
>
>
> On 9/16/11 10:43 AM, Alexander Klenner wrote:
>> Hello,
>>
>> I have a question concerning the merging of different UIMA pipelines. Say I have 3 different annotators that work on the same document (The CAS sofa data is identical for each of the pipelines) They do this parallel and all of them produce different annotations but in a sofa with the same name(_textView). Finally I have 3 serialized XCAS files in three different folders, coming from different nodes of a cluster.
>>
>> Is there an UIMA conform way to merge the corresponding xml files into one CAS object that has all the annotations of the three separate files? I could easily do this with a non uima java class that just adds all the annotation information into one file. Since the sofa data is the same, the offset information of the annotations will be correct, but I'd rather stay in the UIMA context.
>>
>> Cheers,
>>
>> Alex
>>
>> --
>> Dipl. Bioinformatiker Alexander G. Klenner
>> Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI)
>> Schloss Birlinghoven, D-53754 Sankt Augustin
>> Tel.: +49 - 2241 - 14 - 2736
>> E-mail: alexander.garvin.klenner@scai.fraunhofer.de
>> Internet: http://www.scai.fraunhofer.de
>>
>>
>
--
Christophe Roeder | Software Engineer
University of Colorado Anschutz Medical Campus | Computational Bioscience Program
303-724-7562 | chris.roeder@ucdenver.edu | http://compbio.ucdenver.edu
Re: merging CAS objects
Posted by Jörn Kottmann <ko...@gmail.com>.
On 9/16/11 12:12 PM, Alexander Klenner wrote:
> Hi Jörn,
>
> we are using a combination of UNICORE and UIMA, I think it'd take too long to explain the complete architecture but in a nutshell: unicore does all the workflow stuff (passing XCAS obejcts to next job and so on) and all our unicore "grid beans" are independent uima applications (XCAS in - AE - XCAS out), so far it works really fine but at some point we have to merge our CAS objects. The nice thing about this is we can use the unicore rich client to drag and drop any uima AEs into the workflow. It's like a GUI for UIMA AEs now. We have a little test pipeline with 60 diverse documents (8 million characters) and we have some simple and also some complex AEs that need 180 minutes if all AEs are single threaded, this can be reduced to 60 minutes with multithreaded AEs (4 instances each) and it goes down to 20 minutes if we let them do the annotation at the same time using 4 threads each. So it actually does make a difference if the annotation is done one after another or at the same time.
>
I see, it is because you can utilize more cores this way with a small
set of text items. I assumed you have thousands or millions
of text items.
Jörn
Re: merging CAS objects
Posted by Alexander Klenner <al...@scai.fraunhofer.de>.
Hi Jörn,
we are using a combination of UNICORE and UIMA, I think it'd take too long to explain the complete architecture but in a nutshell: unicore does all the workflow stuff (passing XCAS obejcts to next job and so on) and all our unicore "grid beans" are independent uima applications (XCAS in - AE - XCAS out), so far it works really fine but at some point we have to merge our CAS objects. The nice thing about this is we can use the unicore rich client to drag and drop any uima AEs into the workflow. It's like a GUI for UIMA AEs now. We have a little test pipeline with 60 diverse documents (8 million characters) and we have some simple and also some complex AEs that need 180 minutes if all AEs are single threaded, this can be reduced to 60 minutes with multithreaded AEs (4 instances each) and it goes down to 20 minutes if we let them do the annotation at the same time using 4 threads each. So it actually does make a difference if the annotation is done one after another or at the same time.
Cheers,
Alex
--
Dipl. Bioinformatiker Alexander G. Klenner
Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI)
Schloss Birlinghoven, D-53754 Sankt Augustin
Tel.: +49 - 2241 - 14 - 2736
E-mail: alexander.garvin.klenner@scai.fraunhofer.de
Internet: http://www.scai.fraunhofer.de
----- Original Message -----
From: "Jörn Kottmann" <ko...@gmail.com>
To: user@uima.apache.org
Sent: Friday, September 16, 2011 11:48:17 AM
Subject: Re: merging CAS objects
Hello,
depending on your application it might be just as fast if you just have
one CAS and then put it
through your three analysis pipelines, one after another. When you do it
this way your throughput can be just
as high, only the latency goes up.
When you have multiple machines or a cluster you might want to use
UIMA-AS. As far as I know
you can use a parallel step in the flow, which also does the merging for
you.
Jörn
On 9/16/11 10:43 AM, Alexander Klenner wrote:
> Hello,
>
> I have a question concerning the merging of different UIMA pipelines. Say I have 3 different annotators that work on the same document (The CAS sofa data is identical for each of the pipelines) They do this parallel and all of them produce different annotations but in a sofa with the same name(_textView). Finally I have 3 serialized XCAS files in three different folders, coming from different nodes of a cluster.
>
> Is there an UIMA conform way to merge the corresponding xml files into one CAS object that has all the annotations of the three separate files? I could easily do this with a non uima java class that just adds all the annotation information into one file. Since the sofa data is the same, the offset information of the annotations will be correct, but I'd rather stay in the UIMA context.
>
> Cheers,
>
> Alex
>
> --
> Dipl. Bioinformatiker Alexander G. Klenner
> Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI)
> Schloss Birlinghoven, D-53754 Sankt Augustin
> Tel.: +49 - 2241 - 14 - 2736
> E-mail: alexander.garvin.klenner@scai.fraunhofer.de
> Internet: http://www.scai.fraunhofer.de
>
>
Re: merging CAS objects
Posted by Jörn Kottmann <ko...@gmail.com>.
Hello,
depending on your application it might be just as fast if you just have
one CAS and then put it
through your three analysis pipelines, one after another. When you do it
this way your throughput can be just
as high, only the latency goes up.
When you have multiple machines or a cluster you might want to use
UIMA-AS. As far as I know
you can use a parallel step in the flow, which also does the merging for
you.
Jörn
On 9/16/11 10:43 AM, Alexander Klenner wrote:
> Hello,
>
> I have a question concerning the merging of different UIMA pipelines. Say I have 3 different annotators that work on the same document (The CAS sofa data is identical for each of the pipelines) They do this parallel and all of them produce different annotations but in a sofa with the same name(_textView). Finally I have 3 serialized XCAS files in three different folders, coming from different nodes of a cluster.
>
> Is there an UIMA conform way to merge the corresponding xml files into one CAS object that has all the annotations of the three separate files? I could easily do this with a non uima java class that just adds all the annotation information into one file. Since the sofa data is the same, the offset information of the annotations will be correct, but I'd rather stay in the UIMA context.
>
> Cheers,
>
> Alex
>
> --
> Dipl. Bioinformatiker Alexander G. Klenner
> Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI)
> Schloss Birlinghoven, D-53754 Sankt Augustin
> Tel.: +49 - 2241 - 14 - 2736
> E-mail: alexander.garvin.klenner@scai.fraunhofer.de
> Internet: http://www.scai.fraunhofer.de
>
>
Re: merging CAS objects
Posted by Alexander Klenner <al...@scai.fraunhofer.de>.
Hi Richard,
thank you, that sounds good. I'll give it a try.
Cheers,
Alex
--
Dipl. Bioinformatiker Alexander G. Klenner
Fraunhofer-Institute for Algorithms and Scientific Computing (SCAI)
Schloss Birlinghoven, D-53754 Sankt Augustin
Tel.: +49 - 2241 - 14 - 2736
E-mail: alexander.garvin.klenner@scai.fraunhofer.de
Internet: http://www.scai.fraunhofer.de
----- Original Message -----
From: "Richard Eckart de Castilho" <ec...@tk.informatik.tu-darmstadt.de>
To: user@uima.apache.org
Sent: Friday, September 16, 2011 11:15:33 AM
Subject: Re: merging CAS objects
Hello Alex,
you could consider using a CasMultiplier. The name is misleading in this case, it can also function as a "merger".
The principle is simple. It receives CASes and it produces CASes but it can decide to produce multiple output CASes for single input CAS or it can choose to produce an output CAS only in certain cases. The the latter case it can server as a "merger". There would need to be some metadata (some dedicated FeatureStructure) in the CAS that your CasMultiplier-based component can use to decide if a output CAS should be created.
Cheers,
Richard
Am 16.09.2011 um 10:43 schrieb Alexander Klenner:
> Hello,
>
> I have a question concerning the merging of different UIMA pipelines. Say I have 3 different annotators that work on the same document (The CAS sofa data is identical for each of the pipelines) They do this parallel and all of them produce different annotations but in a sofa with the same name(_textView). Finally I have 3 serialized XCAS files in three different folders, coming from different nodes of a cluster.
>
> Is there an UIMA conform way to merge the corresponding xml files into one CAS object that has all the annotations of the three separate files? I could easily do this with a non uima java class that just adds all the annotation information into one file. Since the sofa data is the same, the offset information of the annotations will be correct, but I'd rather stay in the UIMA context.
>
> Cheers,
>
> Alex
--
-------------------------------------------------------------------
Richard Eckart de Castilho
Technical Lead
Ubiquitous Knowledge Processing Lab
FB 20 Computer Science Department
Technische Universität Darmstadt
Hochschulstr. 10, D-64289 Darmstadt, Germany
phone [+49] (0)6151 16-7477, fax -5455, room S2/02/B117
eckartde@tk.informatik.tu-darmstadt.de
www.ukp.tu-darmstadt.de
Web Research at TU Darmstadt (WeRC) www.werc.tu-darmstadt.de
-------------------------------------------------------------------
Re: merging CAS objects
Posted by Richard Eckart de Castilho <ec...@tk.informatik.tu-darmstadt.de>.
Hello Alex,
you could consider using a CasMultiplier. The name is misleading in this case, it can also function as a "merger".
The principle is simple. It receives CASes and it produces CASes but it can decide to produce multiple output CASes for single input CAS or it can choose to produce an output CAS only in certain cases. The the latter case it can server as a "merger". There would need to be some metadata (some dedicated FeatureStructure) in the CAS that your CasMultiplier-based component can use to decide if a output CAS should be created.
Cheers,
Richard
Am 16.09.2011 um 10:43 schrieb Alexander Klenner:
> Hello,
>
> I have a question concerning the merging of different UIMA pipelines. Say I have 3 different annotators that work on the same document (The CAS sofa data is identical for each of the pipelines) They do this parallel and all of them produce different annotations but in a sofa with the same name(_textView). Finally I have 3 serialized XCAS files in three different folders, coming from different nodes of a cluster.
>
> Is there an UIMA conform way to merge the corresponding xml files into one CAS object that has all the annotations of the three separate files? I could easily do this with a non uima java class that just adds all the annotation information into one file. Since the sofa data is the same, the offset information of the annotations will be correct, but I'd rather stay in the UIMA context.
>
> Cheers,
>
> Alex
--
-------------------------------------------------------------------
Richard Eckart de Castilho
Technical Lead
Ubiquitous Knowledge Processing Lab
FB 20 Computer Science Department
Technische Universität Darmstadt
Hochschulstr. 10, D-64289 Darmstadt, Germany
phone [+49] (0)6151 16-7477, fax -5455, room S2/02/B117
eckartde@tk.informatik.tu-darmstadt.de
www.ukp.tu-darmstadt.de
Web Research at TU Darmstadt (WeRC) www.werc.tu-darmstadt.de
-------------------------------------------------------------------
Re: merging CAS objects
Posted by Jens Grivolla <j+...@grivolla.net>.
On 09/16/2011 10:43 AM, Alexander Klenner wrote:
> I have a question concerning the merging of different UIMA pipelines.
> Say I have 3 different annotators that work on the same document (The
> CAS sofa data is identical for each of the pipelines) They do this
> parallel and all of them produce different annotations but in a sofa
> with the same name(_textView). Finally I have 3 serialized XCAS files
> in three different folders, coming from different nodes of a
> cluster.
We have the same problem sometimes, and I'd be very interested in a
"clean" solution.
> Is there an UIMA conform way to merge the corresponding xml files
> into one CAS object that has all the annotations of the three
> separate files? I could easily do this with a non uima java class
> that just adds all the annotation information into one file. Since
> the sofa data is the same, the offset information of the annotations
> will be correct, but I'd rather stay in the UIMA context.
We actually edit XMI files using python scripts to add annotations that
come from outside UIMA, etc. However, especially given the very
unfortunate disappearance of Ed Loper's uimapy, our approach is a bit
hacky, e.g. for dealing with the xmi:id features, namespace prefixes for
type systems, etc. Also, XMI allows for many different representations
of the same information, and our scripts really only deal with the most
common version (as attributes).
I guess in Java you can at least use
org.apache.uima.cas.impl.XmiCasDeserializer and
org.apache.uima.cas.impl.XmiCasSerializer to avoid the XMI specific details.
Bye,
Jens