You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by sethah <gi...@git.apache.org> on 2016/03/30 07:00:29 UTC

[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

GitHub user sethah opened a pull request:

    https://github.com/apache/spark/pull/12056

    [SPARK-14264][PYSPARK][ML] Add feature importance for GBTs in pyspark

    ## What changes were proposed in this pull request?
    
    Feature importances are exposed in the python API for GBTs.
    
    Other changes:
    * Update the random forest feature importance documentation to not repeat decision tree docstring and instead place a reference to it.
    
    
    ## How was this patch tested?
    
    Python doc tests were updated to validate GBT feature importance.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/sethah/spark Pyspark_GBT_feature_importance

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/12056.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #12056
    
----
commit 892d30123a192cd3796892c0f64a5cf2993e1f09
Author: sethah <se...@gmail.com>
Date:   2016-03-30T04:50:33Z

    adding feature importance to gbts in pyspark

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12056#issuecomment-203614085
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54548/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12056#discussion_r57875497
  
    --- Diff: python/pyspark/ml/classification.py ---
    @@ -500,16 +500,12 @@ def featureImportances(self):
             """
             Estimate of the importance of each feature.
     
    -        This generalizes the idea of "Gini" importance to other losses,
    -        following the explanation of Gini importance from "Random Forests" documentation
    -        by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
    +        Each feature's importance is the average of its importance across all trees in the ensemble
    --- End diff --
    
    Why is the doc changing? Is it incorrect?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12056#discussion_r57906908
  
    --- Diff: python/pyspark/ml/classification.py ---
    @@ -500,16 +500,12 @@ def featureImportances(self):
             """
             Estimate of the importance of each feature.
     
    -        This generalizes the idea of "Gini" importance to other losses,
    -        following the explanation of Gini importance from "Random Forests" documentation
    -        by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
    +        Each feature's importance is the average of its importance across all trees in the ensemble
    --- End diff --
    
    There was some discussion on this [here](https://github.com/apache/spark/pull/11961). Basically, since the random forest and GBT importance is just an average of the importances for single trees, we can just state that here and link to the doc for single trees, which explains how those are computed. Otherwise, we copy/paste the same explanation 6 times.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12056#discussion_r57908437
  
    --- Diff: python/pyspark/ml/classification.py ---
    @@ -500,16 +500,12 @@ def featureImportances(self):
             """
             Estimate of the importance of each feature.
     
    -        This generalizes the idea of "Gini" importance to other losses,
    -        following the explanation of Gini importance from "Random Forests" documentation
    -        by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
    +        Each feature's importance is the average of its importance across all trees in the ensemble
    --- End diff --
    
    Fair enough - though the Scala doc should be updated likewise?
    On Wed, 30 Mar 2016 at 17:21, Seth Hendrickson <no...@github.com>
    wrote:
    
    > In python/pyspark/ml/classification.py
    > <https://github.com/apache/spark/pull/12056#discussion_r57906908>:
    >
    > > @@ -500,16 +500,12 @@ def featureImportances(self):
    > >          """
    > >          Estimate of the importance of each feature.
    > >
    > > -        This generalizes the idea of "Gini" importance to other losses,
    > > -        following the explanation of Gini importance from "Random Forests" documentation
    > > -        by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
    > > +        Each feature's importance is the average of its importance across all trees in the ensemble
    >
    > There was some discussion on this here
    > <https://github.com/apache/spark/pull/11961>. Basically, since the random
    > forest and GBT importance is just an average of the importances for single
    > trees, we can just state that here and link to the doc for single trees,
    > which explains how those are computed. Otherwise, we copy/paste the same
    > explanation 6 times.
    >
    > —
    > You are receiving this because you are subscribed to this thread.
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/spark/pull/12056/files/892d30123a192cd3796892c0f64a5cf2993e1f09#r57906908>
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/12056#issuecomment-204102520
  
    LGTM
    Merging with master
    Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12056#issuecomment-203252624
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/54496/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/12056


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12056#discussion_r57953241
  
    --- Diff: python/pyspark/ml/classification.py ---
    @@ -500,16 +500,12 @@ def featureImportances(self):
             """
             Estimate of the importance of each feature.
     
    -        This generalizes the idea of "Gini" importance to other losses,
    -        following the explanation of Gini importance from "Random Forests" documentation
    -        by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
    +        Each feature's importance is the average of its importance across all trees in the ensemble
    +        The importance vector is normalized to sum to 1. This method is suggested by Hastie et al.
    +        (Hastie, Tibshirani, Friedman. "The Elements of Statistical Learning, 2nd Edition." 2001.)
    +        and follows the implementation from scikit-learn.
     
    -        This feature importance is calculated as follows:
    -         - Average over trees:
    -            - importance(feature j) = sum (over nodes which split on feature j) of the gain,
    -              where gain is scaled by the number of instances passing through node
    -            - Normalize importances for tree to sum to 1.
    -         - Normalize feature importance vector to sum to 1.
    +        .. seealso:: :attr:`DecisionTreeClassificationModel.featureImportances`
    --- End diff --
    
    I tried both locally (with/without), and they both seem to work. But I have seen the `:py:` prefix in other places so I updated it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12056#issuecomment-203252623
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12056#issuecomment-203609774
  
    **[Test build #54548 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54548/consoleFull)** for PR 12056 at commit [`0ccf1ed`](https://github.com/apache/spark/commit/0ccf1ed53109ee750b5acebbe2c89ed80da7eacd).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12056#issuecomment-203613911
  
    **[Test build #54548 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54548/consoleFull)** for PR 12056 at commit [`0ccf1ed`](https://github.com/apache/spark/commit/0ccf1ed53109ee750b5acebbe2c89ed80da7eacd).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12056#discussion_r57951739
  
    --- Diff: python/pyspark/ml/classification.py ---
    @@ -500,16 +500,12 @@ def featureImportances(self):
             """
             Estimate of the importance of each feature.
     
    -        This generalizes the idea of "Gini" importance to other losses,
    -        following the explanation of Gini importance from "Random Forests" documentation
    -        by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
    +        Each feature's importance is the average of its importance across all trees in the ensemble
    +        The importance vector is normalized to sum to 1. This method is suggested by Hastie et al.
    +        (Hastie, Tibshirani, Friedman. "The Elements of Statistical Learning, 2nd Edition." 2001.)
    +        and follows the implementation from scikit-learn.
     
    -        This feature importance is calculated as follows:
    -         - Average over trees:
    -            - importance(feature j) = sum (over nodes which split on feature j) of the gain,
    -              where gain is scaled by the number of instances passing through node
    -            - Normalize importances for tree to sum to 1.
    -         - Normalize feature importance vector to sum to 1.
    +        .. seealso:: :attr:`DecisionTreeClassificationModel.featureImportances`
    --- End diff --
    
    Does this need to be ```:py:attr:```?  (same for other places)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12056#issuecomment-203614079
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12056#issuecomment-203252561
  
    **[Test build #54496 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54496/consoleFull)** for PR 12056 at commit [`892d301`](https://github.com/apache/spark/commit/892d30123a192cd3796892c0f64a5cf2993e1f09).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by MLnick <gi...@git.apache.org>.
Github user MLnick commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12056#discussion_r57912592
  
    --- Diff: python/pyspark/ml/classification.py ---
    @@ -500,16 +500,12 @@ def featureImportances(self):
             """
             Estimate of the importance of each feature.
     
    -        This generalizes the idea of "Gini" importance to other losses,
    -        following the explanation of Gini importance from "Random Forests" documentation
    -        by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
    +        Each feature's importance is the average of its importance across all trees in the ensemble
    --- End diff --
    
    Ah sorry I missed that.
    On Wed, 30 Mar 2016 at 17:32, Seth Hendrickson <no...@github.com>
    wrote:
    
    > In python/pyspark/ml/classification.py
    > <https://github.com/apache/spark/pull/12056#discussion_r57909234>:
    >
    > > @@ -500,16 +500,12 @@ def featureImportances(self):
    > >          """
    > >          Estimate of the importance of each feature.
    > >
    > > -        This generalizes the idea of "Gini" importance to other losses,
    > > -        following the explanation of Gini importance from "Random Forests" documentation
    > > -        by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
    > > +        Each feature's importance is the average of its importance across all trees in the ensemble
    >
    > It was update in the PR I linked to above, so everything should be in sync.
    >
    > —
    > You are receiving this because you are subscribed to this thread.
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/spark/pull/12056/files/892d30123a192cd3796892c0f64a5cf2993e1f09#r57909234>
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by sethah <gi...@git.apache.org>.
Github user sethah commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12056#discussion_r57909234
  
    --- Diff: python/pyspark/ml/classification.py ---
    @@ -500,16 +500,12 @@ def featureImportances(self):
             """
             Estimate of the importance of each feature.
     
    -        This generalizes the idea of "Gini" importance to other losses,
    -        following the explanation of Gini importance from "Random Forests" documentation
    -        by Leo Breiman and Adele Cutler, and following the implementation from scikit-learn.
    +        Each feature's importance is the average of its importance across all trees in the ensemble
    --- End diff --
    
    It was update in the PR I linked to above, so everything should be in sync.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14264][PYSPARK][ML] Add feature importa...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12056#issuecomment-203249760
  
    **[Test build #54496 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/54496/consoleFull)** for PR 12056 at commit [`892d301`](https://github.com/apache/spark/commit/892d30123a192cd3796892c0f64a5cf2993e1f09).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org