You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Benson Margulies <bi...@gmail.com> on 2009/05/30 01:24:22 UTC
Text pipeline, or what comes before (ids on vectors)?
I think I have a grip on this, but I'm not quite sure. Please forgive me if
I'm confused and this belongs back on 'users'.
Drivers like the K-Means driver eat vectors and emit clusters of vectors.
The vectors don't carry any sort of field that can be used to back-associate
them with documents or other 'original sources'. These vectors are likely to
be unique, but there's no guarantee.
So, I think that one of Grant's email messages was referring to this problem
(in mentioning that the author of MAHOUT-126 was using a juiced-up mahout
with some solution to this).
Now, before looking to introduce a comprehensive document pipeline, it seems
that what is lacking here is this traceback.
On the one hand, it seems a bit peculiar to propose that Vector object carry
an ID string. On the other hand, the alternative would seem to be some
source of an object that contains a vector and an ID, and changes to (at
least) all the clustering drivers to keep track of which vector (by ID) ends
up in which cluster.
What's the current state of thought?
Re: Text pipeline, or what comes before (ids on vectors)?
Posted by Grant Ingersoll <gs...@apache.org>.
FYI: https://issues.apache.org/jira/browse/MAHOUT-65
On May 29, 2009, at 7:59 PM, Ted Dunning wrote:
> This is a great thing to have.
>
> In general, I think that rows and columns of matrices should be
> labeled. A
> vector is either a row or a column and thus it should have a label
> and its
> elements should have labels. It should not cost you anything if you
> don't
> want labels and I do.
>
> Tracing back to a document id or tracing back to terms or other
> features is
> just much too important to not have this.
>
> My own inspiration for this kind of design is largely based on R's
> data.frame.
>
> On Fri, May 29, 2009 at 4:24 PM, Benson Margulies <bimargulies@gmail.com
> >wrote:
>
>> I think I have a grip on this, but I'm not quite sure. Please
>> forgive me if
>> I'm confused and this belongs back on 'users'.
>>
>> Drivers like the K-Means driver eat vectors and emit clusters of
>> vectors.
>> The vectors don't carry any sort of field that can be used to
>> back-associate
>> them with documents or other 'original sources'. These vectors are
>> likely
>> to
>> be unique, but there's no guarantee.
>>
>> So, I think that one of Grant's email messages was referring to this
>> problem
>> (in mentioning that the author of MAHOUT-126 was using a juiced-up
>> mahout
>> with some solution to this).
>>
>> Now, before looking to introduce a comprehensive document pipeline,
>> it
>> seems
>> that what is lacking here is this traceback.
>>
>> On the one hand, it seems a bit peculiar to propose that Vector
>> object
>> carry
>> an ID string. On the other hand, the alternative would seem to be
>> some
>> source of an object that contains a vector and an ID, and changes
>> to (at
>> least) all the clustering drivers to keep track of which vector (by
>> ID)
>> ends
>> up in which cluster.
>>
>> What's the current state of thought?
>>
>
>
>
> --
> Ted Dunning, CTO
> DeepDyve
>
> 111 West Evelyn Ave. Ste. 202
> Sunnyvale, CA 94086
> http://www.deepdyve.com
> 858-414-0013 (m)
> 408-773-0220 (fax)
--------------------------
Grant Ingersoll
http://www.lucidimagination.com/
Search the Lucene ecosystem (Lucene/Solr/Nutch/Mahout/Tika/Droids)
using Solr/Lucene:
http://www.lucidimagination.com/search
Re: Text pipeline, or what comes before (ids on vectors)?
Posted by Ted Dunning <te...@gmail.com>.
This is a great thing to have.
In general, I think that rows and columns of matrices should be labeled. A
vector is either a row or a column and thus it should have a label and its
elements should have labels. It should not cost you anything if you don't
want labels and I do.
Tracing back to a document id or tracing back to terms or other features is
just much too important to not have this.
My own inspiration for this kind of design is largely based on R's
data.frame.
On Fri, May 29, 2009 at 4:24 PM, Benson Margulies <bi...@gmail.com>wrote:
> I think I have a grip on this, but I'm not quite sure. Please forgive me if
> I'm confused and this belongs back on 'users'.
>
> Drivers like the K-Means driver eat vectors and emit clusters of vectors.
> The vectors don't carry any sort of field that can be used to
> back-associate
> them with documents or other 'original sources'. These vectors are likely
> to
> be unique, but there's no guarantee.
>
> So, I think that one of Grant's email messages was referring to this
> problem
> (in mentioning that the author of MAHOUT-126 was using a juiced-up mahout
> with some solution to this).
>
> Now, before looking to introduce a comprehensive document pipeline, it
> seems
> that what is lacking here is this traceback.
>
> On the one hand, it seems a bit peculiar to propose that Vector object
> carry
> an ID string. On the other hand, the alternative would seem to be some
> source of an object that contains a vector and an ID, and changes to (at
> least) all the clustering drivers to keep track of which vector (by ID)
> ends
> up in which cluster.
>
> What's the current state of thought?
>
--
Ted Dunning, CTO
DeepDyve
111 West Evelyn Ave. Ste. 202
Sunnyvale, CA 94086
http://www.deepdyve.com
858-414-0013 (m)
408-773-0220 (fax)