You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Deejay <de...@binarytweed.com> on 2012/06/08 17:44:47 UTC

Clustering, Collapsing

Hi all,

I recently discovered Apache UIMA, and it looks like a very large project! I 
was hoping that someone more experienced with it than I could comment on 
whether there are parts of the project that could help with my problem.

I need to go over many millions of objects (Protocol Buffers in HBase, as it
happens), and cluster them according to their similarity. Once each cluster is
formed, I need to 'collapse' each property of the objects to find the most 
prevalent value. After this, the collapsed object will be added to a Solr 
index.

Would any part of Apache UIMA be useful for the clustering or collapsing, or 
have I misunderstood the nature of the project?


Re: Clustering, Collapsing

Posted by Deejay <de...@binarytweed.com>.
Hi Tommaso,

Thanks for the reply! 

> Apart from the specific algorithms used for clustering / collapsing, which
> would define the UIMA pipeline implementations/configurations, you could
> use SolrCas [3] to finally write data in the index.

Are there algorithm implementations built into the UIMA project that could be 
used as-is or tweaked, or would we need to create these from scratch?

We have quite a large investment in the Hadoop ecosystem. Is it a common 
use-case to parallelise UIMA jobs using map/reduce?


Re: Clustering, Collapsing

Posted by Tommaso Teofili <to...@gmail.com>.
Hi Deejay,

2012/6/8 Deejay <de...@binarytweed.com>

> Hi all,
>
> I recently discovered Apache UIMA, and it looks like a very large project!
> I
> was hoping that someone more experienced with it than I could comment on
> whether there are parts of the project that could help with my problem.
>
> I need to go over many millions of objects (Protocol Buffers in HBase, as
> it
> happens), and cluster them according to their similarity. Once each
> cluster is
> formed, I need to 'collapse' each property of the objects to find the most
> prevalent value. After this, the collapsed object will be added to a Solr
> index.
>

I think you could take advantage of UIMA Collection Processing Engine [1],
particularly by using a UIMA-AS based architecture since it looks like you
are handling huge collections [2].
Apart from the specific algorithms used for clustering / collapsing, which
would define the UIMA pipeline implementations/configurations, you could
use SolrCas [3] to finally write data in the index.


>
> Would any part of Apache UIMA be useful for the clustering or collapsing,
> or
> have I misunderstood the nature of the project?
>
>
HTH
Tommaso

[1] :
http://uima.apache.org/d/uimaj-2.4.0/tutorials_and_users_guides.html#ugr.tug.cpe
[2] : http://uima.apache.org/doc-uimaas-what.html
[3] : http://uima.apache.org/sandbox.html#solrcas.consumer

Re: Clustering, Collapsing

Posted by Jens Grivolla <j+...@grivolla.net>.
This sounds like you are actually looking for the project next door: Mahout.

UIMA really doesn't have a lot to do with clustering (although you could 
do some things). We do use UIMA for information extraction *before* 
clustering and sending it to Solr, though, as a sort of preprocessing to 
get relevant features from unstructured text. But it doesn't sound like 
that's what you're trying to do.

HTH,
Jens

On 06/08/2012 05:44 PM, Deejay wrote:
> Hi all,
>
> I recently discovered Apache UIMA, and it looks like a very large project! I
> was hoping that someone more experienced with it than I could comment on
> whether there are parts of the project that could help with my problem.
>
> I need to go over many millions of objects (Protocol Buffers in HBase, as it
> happens), and cluster them according to their similarity. Once each cluster is
> formed, I need to 'collapse' each property of the objects to find the most
> prevalent value. After this, the collapsed object will be added to a Solr
> index.
>
> Would any part of Apache UIMA be useful for the clustering or collapsing, or
> have I misunderstood the nature of the project?
>
>