You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@uima.apache.org by Deejay <de...@binarytweed.com> on 2012/06/08 17:44:47 UTC
Clustering, Collapsing
Hi all,
I recently discovered Apache UIMA, and it looks like a very large project! I
was hoping that someone more experienced with it than I could comment on
whether there are parts of the project that could help with my problem.
I need to go over many millions of objects (Protocol Buffers in HBase, as it
happens), and cluster them according to their similarity. Once each cluster is
formed, I need to 'collapse' each property of the objects to find the most
prevalent value. After this, the collapsed object will be added to a Solr
index.
Would any part of Apache UIMA be useful for the clustering or collapsing, or
have I misunderstood the nature of the project?
Re: Clustering, Collapsing
Posted by Deejay <de...@binarytweed.com>.
Hi Tommaso,
Thanks for the reply!
> Apart from the specific algorithms used for clustering / collapsing, which
> would define the UIMA pipeline implementations/configurations, you could
> use SolrCas [3] to finally write data in the index.
Are there algorithm implementations built into the UIMA project that could be
used as-is or tweaked, or would we need to create these from scratch?
We have quite a large investment in the Hadoop ecosystem. Is it a common
use-case to parallelise UIMA jobs using map/reduce?
Re: Clustering, Collapsing
Posted by Tommaso Teofili <to...@gmail.com>.
Hi Deejay,
2012/6/8 Deejay <de...@binarytweed.com>
> Hi all,
>
> I recently discovered Apache UIMA, and it looks like a very large project!
> I
> was hoping that someone more experienced with it than I could comment on
> whether there are parts of the project that could help with my problem.
>
> I need to go over many millions of objects (Protocol Buffers in HBase, as
> it
> happens), and cluster them according to their similarity. Once each
> cluster is
> formed, I need to 'collapse' each property of the objects to find the most
> prevalent value. After this, the collapsed object will be added to a Solr
> index.
>
I think you could take advantage of UIMA Collection Processing Engine [1],
particularly by using a UIMA-AS based architecture since it looks like you
are handling huge collections [2].
Apart from the specific algorithms used for clustering / collapsing, which
would define the UIMA pipeline implementations/configurations, you could
use SolrCas [3] to finally write data in the index.
>
> Would any part of Apache UIMA be useful for the clustering or collapsing,
> or
> have I misunderstood the nature of the project?
>
>
HTH
Tommaso
[1] :
http://uima.apache.org/d/uimaj-2.4.0/tutorials_and_users_guides.html#ugr.tug.cpe
[2] : http://uima.apache.org/doc-uimaas-what.html
[3] : http://uima.apache.org/sandbox.html#solrcas.consumer
Re: Clustering, Collapsing
Posted by Jens Grivolla <j+...@grivolla.net>.
This sounds like you are actually looking for the project next door: Mahout.
UIMA really doesn't have a lot to do with clustering (although you could
do some things). We do use UIMA for information extraction *before*
clustering and sending it to Solr, though, as a sort of preprocessing to
get relevant features from unstructured text. But it doesn't sound like
that's what you're trying to do.
HTH,
Jens
On 06/08/2012 05:44 PM, Deejay wrote:
> Hi all,
>
> I recently discovered Apache UIMA, and it looks like a very large project! I
> was hoping that someone more experienced with it than I could comment on
> whether there are parts of the project that could help with my problem.
>
> I need to go over many millions of objects (Protocol Buffers in HBase, as it
> happens), and cluster them according to their similarity. Once each cluster is
> formed, I need to 'collapse' each property of the objects to find the most
> prevalent value. After this, the collapsed object will be added to a Solr
> index.
>
> Would any part of Apache UIMA be useful for the clustering or collapsing, or
> have I misunderstood the nature of the project?
>
>