Multi-Document Processing

[Matthew Campbell <mc...@syrres.com>]

Hey folks:

    I'm looking at a process that runs each document through a bunch of 
annotators to tag up various information, then I need to do some 
processing/manipulation of those documents based the information held in 
the whole collection.  I've been reading up on the CPE, but it looks 
like it's primarily for running a collection of documents through an 
AE.  I was hoping someone could point me in the right direction for 
doing the collection-wide processing portion of my process.

    I had started out by defining the process as one large aggregate AE 
and running each document through it, but I don't see a way to go 
through that initial tagging process for all documents and then move on 
to the next phase.
    I then switched gears and tried splitting up each phase into it's 
own AE, but then I loose the complex Sofa mappings I had put together 
for the previous attempt.  So I guess this could be solved in two ways - 
one would be that the CPE has some sort of built-in method for doing 
collection-wide processing and manipulation (ie, "first identify all 
location names in all documents, then replace each with a new name, but 
make sure the new name doesn't appear in any other document").  The 
other would be to somehow run through the first phase to identify 
everything, do processing using the collection of JCas's resulting, then 
pump each JCas into a second AE for doing post-processing stuff.  
Somewhere in there would have to be some dynamically-mapped Sofas from 
the phase 1 AE to the phase 2 AE.

    I hope that described my goal well enough, and thanks ahead of time 
for any pointers you guys can throw my way.


-Matt

Re: Multi-Document Processing

[Matthew Campbell <mc...@syrres.com>]

Thanks so much!  That does help - I'm still fiddling with making sure my 
various Sofa's are getting through alright, but this gets me in the 
right direction.


-Matt

Marshall Schor wrote:
> Matthew Campbell wrote:
>> Hey folks:
>>
>>    I'm looking at a process that runs each document through a bunch 
>> of annotators to tag up various information, then I need to do some 
>> processing/manipulation of those documents based the information held 
>> in the whole collection.  I've been reading up on the CPE, but it 
>> looks like it's primarily for running a collection of documents 
>> through an AE.  I was hoping someone could point me in the right 
>> direction for doing the collection-wide processing portion of my 
>> process.
>>    I had started out by defining the process as one large aggregate 
>> AE and running each document through it, but I don't see a way to go 
>> through that initial tagging process for all documents and then move 
>> on to the next phase.
>>    I then switched gears and tried splitting up each phase into it's 
>> own AE, but then I loose the complex Sofa mappings I had put together 
>> for the previous attempt.  So I guess this could be solved in two 
>> ways - one would be that the CPE has some sort of built-in method for 
>> doing collection-wide processing and manipulation (ie, "first 
>> identify all location names in all documents, then replace each with 
>> a new name, but make sure the new name doesn't appear in any other 
>> document").  The other would be to somehow run through the first 
>> phase to identify everything, do processing using the collection of 
>> JCas's resulting, then pump each JCas into a second AE for doing 
>> post-processing stuff.  Somewhere in there would have to be some 
>> dynamically-mapped Sofas from the phase 1 AE to the phase 2 AE.
>>
>>    I hope that described my goal well enough, and thanks ahead of 
>> time for any pointers you guys can throw my way.
>>
> The way many do things like this is to have a singleton Annotator at 
> the end of the pipe line, which sees all of the CASes being processed 
> after they've been "tagged" by earlier annotators.  This annotator 
> would have some persistent Java object(s) that accumulated information 
> across the entire document collection, and would have a 
> collection-processing-complete method which it would register with the 
> CPM so it could be called at the end of processing the collection.  
> This method would then use the accumulated information to do whatever 
> processing you wanted to do at that point.
>
> Would that work?
> -Marshall
>

Re: Multi-Document Processing

[Marshall Schor <ms...@schor.com>]

Matthew Campbell wrote:
> Hey folks:
>
>    I'm looking at a process that runs each document through a bunch of 
> annotators to tag up various information, then I need to do some 
> processing/manipulation of those documents based the information held 
> in the whole collection.  I've been reading up on the CPE, but it 
> looks like it's primarily for running a collection of documents 
> through an AE.  I was hoping someone could point me in the right 
> direction for doing the collection-wide processing portion of my process.
>    I had started out by defining the process as one large aggregate AE 
> and running each document through it, but I don't see a way to go 
> through that initial tagging process for all documents and then move 
> on to the next phase.
>    I then switched gears and tried splitting up each phase into it's 
> own AE, but then I loose the complex Sofa mappings I had put together 
> for the previous attempt.  So I guess this could be solved in two ways 
> - one would be that the CPE has some sort of built-in method for doing 
> collection-wide processing and manipulation (ie, "first identify all 
> location names in all documents, then replace each with a new name, 
> but make sure the new name doesn't appear in any other document").  
> The other would be to somehow run through the first phase to identify 
> everything, do processing using the collection of JCas's resulting, 
> then pump each JCas into a second AE for doing post-processing stuff.  
> Somewhere in there would have to be some dynamically-mapped Sofas from 
> the phase 1 AE to the phase 2 AE.
>
>    I hope that described my goal well enough, and thanks ahead of time 
> for any pointers you guys can throw my way.
>
The way many do things like this is to have a singleton Annotator at the 
end of the pipe line, which sees all of the CASes being processed after 
they've been "tagged" by earlier annotators.  This annotator would have 
some persistent Java object(s) that accumulated information across the 
entire document collection, and would have a 
collection-processing-complete method which it would register with the 
CPM so it could be called at the end of processing the collection.  This 
method would then use the accumulated information to do whatever 
processing you wanted to do at that point.

Would that work?
-Marshall