You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Deneche A. Hakim (JIRA)" <ji...@apache.org> on 2009/12/10 20:58:18 UTC
[jira] Commented: (MAHOUT-216) Improve the results of MAHOUT-145 by
uniformly distributing the classes in the partitioned data
[ https://issues.apache.org/jira/browse/MAHOUT-216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788878#action_12788878 ]
Deneche A. Hakim commented on MAHOUT-216:
-----------------------------------------
First of all I implemented a simple mapreduce tool that computes the frequency distribution of the "class attribute" over the partitions.
Applying this tool over KDD 100% (the whole dataset) I got the following results:
||||||label1||label2||||label3||...||
|partition 0|380275| 3| 1| 1| 15| 112574| 53| 20| 99| 405| 1948| 1| 6| 1000| 1| 24| 1| 1040| 0| 0| 0| 0| 0|
|partition 1|182112| 2| 1| 1| 204800| 95158| 0| 20| 100| 2377| 5631| 16| 2| 1002| 11| 5365| 2| 1276| 6| 20| 0| 0| 0|
|partition 2|149880| 7| 6| 0| 206400| 124516| 0| 62| 198| 3609| 1023| 0| 0| 101| 0| 10489| 0| 0| 1| 0| 1020| 2| 7|
|partition 3|0| 0| 0| 0| 0| 489478| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
|partition 4|0| 0| 0| 0| 0| 489475| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
|partition 5|0| 0| 0| 0| 0| 489476| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
|partition 6|52276| 3| 0| 1| 0| 434768| 0| 41| 99| 1027| 2556| 0| 0| 0| 0| 6| 0| 0| 0| 0| 0| 0| 0|
|partition 7|19629| 2| 1| 0| 449092| 28553| 0| 99| 383| 1191| 9| 1| 0| 0| 0| 4| 0| 0| 0| 0| 0| 0| 1|
|partition 8|0| 0| 0| 0| 0| 492696| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
|partition 9|188609| 13| 0| 0| 211710| 51192| 0| 22| 100| 1804| 1314| 3| 0| 100| 0| 4| 1| 0| 0| 0| 0| 0| 2|
As you can see the classes are not distributed uniformly, some partitions contain only one class !
> Improve the results of MAHOUT-145 by uniformly distributing the classes in the partitioned data
> -----------------------------------------------------------------------------------------------
>
> Key: MAHOUT-216
> URL: https://issues.apache.org/jira/browse/MAHOUT-216
> Project: Mahout
> Issue Type: Improvement
> Components: Classification
> Reporter: Deneche A. Hakim
> Assignee: Deneche A. Hakim
>
> the poor results of the partial decision forest implementation may be explained by the particular distribution of the partitioned data. For example, if a partition does not contain any instance of a given class, the decision trees built using this partition won't be able to classify this class.
> According to [CHAN, 95]:
> {quote}
> Random Selection of the partitioned data sets with a uniform distribution of classes is perhaps the most sensible solution. Here we may attempt to maintain the same frequency distribution over the ''class attribute" so that each partition represents a good but a smaller model of the entire training set
> {quote}
> [CHAN, 95]: Philip K. Chan, "On the Accuracy of Meta-learning for Scalable Data Mining"
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.