[jira] [Created] (SPARK-17645) Add feature selector methods based on: False Discovery Rate (FDR) and Family Wise Error rate (FWE)

["Peng Meng (JIRA)" <ji...@apache.org>]

Peng Meng created SPARK-17645:
---------------------------------

             Summary: Add feature selector methods based on: False Discovery Rate (FDR) and Family Wise Error rate (FWE)
                 Key: SPARK-17645
                 URL: https://issues.apache.org/jira/browse/SPARK-17645
             Project: Spark
          Issue Type: New Feature
          Components: ML, MLlib
            Reporter: Peng Meng
            Priority: Minor


Univariate feature selection works by selecting the best features based on univariate statistical tests. 
FDR and FWE are a popular univariate statistical test for feature selection.

In 2005, the Benjamini and Hochberg paper on FDR was identified as one of the 25 most-cited statistical papers. The FDR uses the Benjamini-Hochberg procedure in this PR. https://en.wikipedia.org/wiki/False_discovery_rate. 
In statistics, FWE is the probability of making one or more false discoveries, or type I errors, among all the hypotheses when performing multiple hypotheses tests.
https://en.wikipedia.org/wiki/Family-wise_error_rate

We add FDR and FWE methods for ChiSqSelector in this PR, like it is implemented in scikit-learn. 
http://scikit-learn.org/stable/modules/feature_selection.html#univariate-feature-selection



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org