You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by "Deneche A. Hakim (JIRA)" <ji...@apache.org> on 2009/12/10 20:58:18 UTC

[jira] Commented: (MAHOUT-216) Improve the results of MAHOUT-145 by uniformly distributing the classes in the partitioned data

    [ https://issues.apache.org/jira/browse/MAHOUT-216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12788878#action_12788878 ] 

Deneche A. Hakim commented on MAHOUT-216:
-----------------------------------------

First of all I implemented a simple mapreduce tool that computes the frequency distribution of the "class attribute" over the partitions.

Applying this tool over KDD 100% (the whole dataset) I got the following results:
||||||label1||label2||||label3||...||
|partition 0|380275|	3|	1|	1|	15|	112574|	53|	20|	99|	405|	1948|	1|	6|	1000|	1|	24|	1|	1040|	0|	0|	0|	0|	0|
|partition 1|182112|	2|	1|	1|	204800|	95158|	0|	20|	100|	2377|	5631|	16|	2|	1002|	11|	5365|	2|	1276|	6|	20|	0|	0|	0|
|partition 2|149880|	7|	6|	0|	206400|	124516|	0|	62|	198|	3609|	1023|	0|	0|	101|	0|	10489|	0|	0|	1|	0|	1020|	2|	7|
|partition 3|0|	0|	0|	0|	0|	489478|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|
|partition 4|0|	0|	0|	0|	0|	489475|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|
|partition 5|0|	0|	0|	0|	0|	489476|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|
|partition 6|52276|	3|	0|	1|	0|	434768|	0|	41|	99|	1027|	2556|	0|	0|	0|	0|	6|	0|	0|	0|	0|	0|	0|	0|
|partition 7|19629|	2|	1|	0|	449092|	28553|	0|	99|	383|	1191|	9|	1|	0|	0|	0|	4|	0|	0|	0|	0|	0|	0|	1|
|partition 8|0|	0|	0|	0|	0|	492696|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|	0|
|partition 9|188609|	13|	0|	0|	211710|	51192|	0|	22|	100|	1804|	1314|	3|	0|	100|	0|	4|	1|	0|	0|	0|	0|	0|	2|

As you can see the classes are not distributed uniformly, some partitions contain only one class !

> Improve the results of MAHOUT-145 by uniformly distributing the classes in the partitioned data
> -----------------------------------------------------------------------------------------------
>
>                 Key: MAHOUT-216
>                 URL: https://issues.apache.org/jira/browse/MAHOUT-216
>             Project: Mahout
>          Issue Type: Improvement
>          Components: Classification
>            Reporter: Deneche A. Hakim
>            Assignee: Deneche A. Hakim
>
> the poor results of the partial decision forest implementation may be explained by the particular distribution of the partitioned data. For example, if a partition does not contain any instance of a given class, the decision trees built using this partition won't be able to classify this class. 
> According to [CHAN, 95]:
> {quote}
> Random Selection of the partitioned data sets with a uniform distribution of classes is perhaps the most sensible solution. Here we may attempt to maintain the same frequency distribution over the ''class attribute" so that each partition represents a good but a smaller model of the entire training set
> {quote}
> [CHAN, 95]: Philip K. Chan, "On the Accuracy of Meta-learning for Scalable Data Mining" 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.