You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Jeff Eastman (JIRA)" <ji...@apache.org> on 2012/06/09 18:09:43 UTC
[jira] [Created] (MAHOUT-1030) Regression: Clustered Points Should
be WeightedPropertyVectorWritable not WeightedVectorWritable
Jeff Eastman created MAHOUT-1030:
------------------------------------
Summary: Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
Key: MAHOUT-1030
URL: https://issues.apache.org/jira/browse/MAHOUT-1030
Project: Mahout
Issue Type: Bug
Components: Clustering, Integration
Affects Versions: 0.7
Reporter: Jeff Eastman
Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
On 6/8/12 12:21 PM, Jeff Eastman wrote:
> That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
>
> On 6/7/12 1:00 PM, Pat Ferrel wrote:
>> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
>>
>> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
>>
>> Am I missing something?
>>
>>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1030) Regression: Clustered Points
Should be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Pat Ferrel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403570#comment-13403570 ]
Pat Ferrel commented on MAHOUT-1030:
------------------------------------
Jeff said; "It is trivial to back-calculate the distance from the pdf value that is already written in the WeightedVectorWritable."
Hmm, I don't see how the pdf is included in the WeightedVectorWritable, at least for kmeans. The weight therein is the probability it belongs to the cluster, for kmeans 1 or 0, right? clusteredPoints/part-m-00000 is made of IntWritable clusterIDs and WeightedVectorWritable so how do you find a non-binary pdf for a specific doc?
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.8
>
> Attachments: MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-1030) Regression: Clustered Points Should
be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-1030:
---------------------------------
Attachment: MAHOUT-1030.patch
Here's a patch that modifies the cluster classification code to emit WeightedPropertyVectorWritables instead of WeightedVectorWritables. It does not compute a distance property; however, as this is not generally available for all Cluster types (really only DistanceMeasureClusters).
The more I think about the distance property calculation the more I am comfortable with the current (trunk) implementation. Consider that, for DistanceMeasureClusters at least, the pdf is:
pdf = 1/(1+distance)
It is trivial to back-calculate the distance from the pdf value that is already written in the WeightedVectorWritable. This is because, in the new implementation, the pdfs are calculated by the classify() policy method and it is the pdfs that are written.
Sooo, I'm not sure this is a must-fix or even a want-fix. Thoughts?
This patch also fixes a bug in the ClusterDumperWriter that correctly catches null properties fields read when the written properties map contains 0 entries. This part of the patch should go into trunk regardless of the above.
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Attachments: MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-1030) Regression: Clustered Points Should
be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-1030:
---------------------------------
Fix Version/s: 0.8
I'm moving this to 0.8 to allow for more discussion on whether or not it needs to be implemented at all. Given that the existing pdf weights allow for distance to be computed I don't view it as a showstopper for 0.7.
I committed the patch to ClusterDumperWriter after the release bits were cut but, since it has not presented itself to date and since clustering no longer produces WeightedPropertyVectorWritables, I don't view it as a showstopper for 0.7 either.
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.8
>
> Attachments: MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1030) Regression: Clustered Points
Should be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Lance Norskog (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404650#comment-13404650 ]
Lance Norskog commented on MAHOUT-1030:
---------------------------------------
bq. If this fixes the regression adequately, it introduces another incompatibility with AbstractVectorClassifier's expectation of normalized results.
I have not used this code, but have a general request: it is hard to describe the difficulty of approaching this kind of code from outside. Inconsistency makes it that much more frustrating.
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.8
>
> Attachments: MAHOUT-1030.patch, MAHOUT-1030.patch, MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1030) Regression: Clustered Points
Should be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Pat Ferrel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404581#comment-13404581 ]
Pat Ferrel commented on MAHOUT-1030:
------------------------------------
Personally I have a work around that's independent of putting the distance or pdf in the clusteredPoints WeightedVectorWritable.
Clearly there are larger issues regarding the refactoring that have to be considered. I'll rely on my work around until the mahout team has determined what should get into the trunk and will then convert to that solution.
If you think this patch is the right thing to do I'll test it.
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.8
>
> Attachments: MAHOUT-1030.patch, MAHOUT-1030.patch, MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-1030) Regression: Clustered Points Should
be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-1030:
---------------------------------
Attachment: MAHOUT-1030.patch
Here's a small patch that actually sets the documented weights in the WeightedVectorWritables in the ClusterClassificationMapper. Needs more testing to make sure it doesn't break something else though.
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.8
>
> Attachments: MAHOUT-1030.patch, MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1030) Regression: Clustered Points
Should be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Pat Ferrel (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404737#comment-13404737 ]
Pat Ferrel commented on MAHOUT-1030:
------------------------------------
A couple other thoughts:
1) With non-partitioning clustering like fuzzy kmeans the weight is used to communicate the probability of inclusion in the cluster. I'd expect it is required. Whatever you do can't change the meaning of the weight for anything but kmeans.
2) Strictly speaking the probability of a vectors inclusion in a cluster for kmeans is always 1.0 or 0. Therefore if you attach the underlying pdf to the clusteredPoints aren't you actually turning kmeans into fkmeans, at least by some loose definition? or is the weight calculated differently for fkmeans--see #1.
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.8
>
> Attachments: MAHOUT-1030.patch, MAHOUT-1030.patch, MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1030) Regression: Clustered Points
Should be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292427#comment-13292427 ]
Hudson commented on MAHOUT-1030:
--------------------------------
Integrated in Mahout-Quality #1537 (See [https://builds.apache.org/job/Mahout-Quality/1537/])
MAHOUT-1030: adding null check when empty properties are written (Revision 1348491)
Result = SUCCESS
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348491
Files :
* /mahout/trunk/integration/src/main/java/org/apache/mahout/utils/clustering/ClusterDumperWriter.java
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-1030) Regression: Clustered Points Should
be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-1030:
---------------------------------
Assignee: Jeff Eastman
Status: Patch Available (was: Open)
Not sure this patch is what we want to do for the reasons mentioned above. I think the current implementation provides enough information already.
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAHOUT-1030) Regression: Clustered Points
Should be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13292438#comment-13292438 ]
Hudson commented on MAHOUT-1030:
--------------------------------
Integrated in Mahout-Quality #1538 (See [https://builds.apache.org/job/Mahout-Quality/1538/])
MAHOUT-1030: adding better null check when empty properties are written (Revision 1348494)
MAHOUT-1030: adding better null check when empty properties are written (Revision 1348493)
Result = SUCCESS
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348494
Files :
* /mahout/trunk/examples/bin/cluster-reuters.sh
* /mahout/trunk/integration/src/main/java/org/apache/mahout/utils/clustering/ClusterDumperWriter.java
jeastman : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1348493
Files :
* /mahout/trunk/integration/src/main/java/org/apache/mahout/utils/clustering/ClusterDumperWriter.java
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Attachments: MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAHOUT-1030) Regression: Clustered Points Should
be WeightedPropertyVectorWritable not WeightedVectorWritable
Posted by "Jeff Eastman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAHOUT-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jeff Eastman updated MAHOUT-1030:
---------------------------------
Attachment: MAHOUT-1030.patch
This patch modifies the classify() method of AbstractClusteringPolicy to remove the normalization in order that pdf values in the WeightedVectorWritable can be back-converted into the original distance measure distance.
Not sure which direction to go on this issue right now. If this fixes the regression adequately, it introduces another incompatibility with AbstractVectorClassifier's expectation of normalized results. The downside of this approach needs to be weighed against the much larger patch needed to return WeightedPropertyVectorWritables in the first version of the patch.
> Regression: Clustered Points Should be WeightedPropertyVectorWritable not WeightedVectorWritable
> ------------------------------------------------------------------------------------------------
>
> Key: MAHOUT-1030
> URL: https://issues.apache.org/jira/browse/MAHOUT-1030
> Project: Mahout
> Issue Type: Bug
> Components: Clustering, Integration
> Affects Versions: 0.7
> Reporter: Jeff Eastman
> Assignee: Jeff Eastman
> Fix For: 0.8
>
> Attachments: MAHOUT-1030.patch, MAHOUT-1030.patch, MAHOUT-1030.patch
>
>
> Looks like this won't make it into this build. Pretty widespread impact on code and tests and I don't know which properties were implemented in the old version. I will create a JIRA and post my interim results.
> On 6/8/12 12:21 PM, Jeff Eastman wrote:
> > That's a reversion that evidently got in when the new ClusterClassificationDriver was introduced. It should be a pretty easy fix and I will see if I can make the change before Paritosh cuts the release bits tonight.
> >
> > On 6/7/12 1:00 PM, Pat Ferrel wrote:
> >> It appears that in kmeans the clusteredPoints are now written as WeightedVectorWritable where in mahout 0.6 they were WeightedPropertyVectorWritable? This means that the distance from the centroid is no longer stored here? Why? I hope I'm wrong because that is not a welcome change. How is one to order clustered docs by distance from cluster centroid?
> >>
> >> I'm sure I could calculate the distance but that would mean looking up the centroid for the cluster id given in the above WeightedVectorWritable, which means iterating through all the clusters for each clustered doc. In my case the number of clusters could be fairly large.
> >>
> >> Am I missing something?
> >>
> >>
> >
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira