You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jkbradley <gi...@git.apache.org> on 2014/11/26 02:05:38 UTC

[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

GitHub user jkbradley opened a pull request:

    https://github.com/apache/spark/pull/3461

    [SPARK-4580] [SPARK-4610] [mllib] Documentation for tree ensembles + DecisionTree API fix

    Major changes:
    * Added documentation for tree ensembles
    * Added examples for tree ensembles
    * **API change**: Standardized the tree parameter for the number of classes (for classification)
    
    Minor changes:
    * Updated decision tree documentation
    * Updated existing tree and tree ensemble examples
     * Use train/test split, and compute test error instead of training error.
     * Fixed decision_tree_runner.py to actually use the number of classes it computes from data. (small bug fix)
    
    Note: I know this is a lot of lines, but most is covered by:
    * Programming guide sections for gradient boosting and random forests.  (The changes are probably best viewed by generating the docs locally.)
    * New examples (which were copied from the programming guide)
    * The "numClasses" renaming
    
    CC: @mengxr @manishamde @codedeft

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/jkbradley/spark ensemble-docs

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/3461.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3461
    
----
commit ad3e6952aabbdabb9f5354eae5e0fec30e9b07f8
Author: Joseph K. Bradley <jo...@databricks.com>
Date:   2014-11-25T18:17:20Z

    added gbt and random forest to programming guide.  still need to update their examples

commit 6372a2b66105d4e5a512f191c3b2518e275d69a2
Author: Joseph K. Bradley <jo...@databricks.com>
Date:   2014-11-25T20:32:06Z

    updated decision tree examples to use random split.  tested all of them.

commit cdfdfbca641c605172171b706c02bdba067466bb
Author: Joseph K. Bradley <jo...@databricks.com>
Date:   2014-11-25T22:58:10Z

    added examples for GBT

commit 07fc11db76c9c38e265fa23957394d521d0bd7eb
Author: Joseph K. Bradley <jo...@databricks.com>
Date:   2014-11-25T23:48:13Z

    Renamed numClassesForClassification to numClasses everywhere in trees and ensembles.
    This is a breaking API change, but it was necessary to correct an API inconsistency in Spark 1.1 (where Python DecisionTree used numClasses but Scala used numClassesForClassification).
    
    Added examples to programming guide for all ensembles.

commit abe5ed7dcfa365cff79e49737658209569b406d7
Author: Joseph K. Bradley <jo...@databricks.com>
Date:   2014-11-26T00:19:03Z

    added examples for random forest in Java and Python to examples folder

commit c76c823ee52ecd08039ea63aff90a732a2e073b8
Author: Joseph K. Bradley <jo...@databricks.com>
Date:   2014-11-26T00:31:41Z

    added migration guide for mllib

commit 706d332b5e9a0ed85518d5ad62e57a9908169028
Author: Joseph K. Bradley <jo...@databricks.com>
Date:   2014-11-26T00:56:39Z

    updated python DT runner to print full model if it is small

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21067775
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    --- End diff --
    
    Could we use 0 -> 2 instead of 1 -> 2 as example? It will avoid confusion about zero-indexing of the feature indices.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-65167028
  
    @jkbradley The GBDT sections looks good to me but the subsection on Comparison with RFs could possibly be moved towards the end. It breaks the flow in my opinion.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64503902
  
      [Test build #23858 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23858/consoleFull) for   PR 3461 at commit [`2b60b6e`](https://github.com/apache/spark/commit/2b60b6efd9284b9cbd7a80d1abbffc9e95d2ff0d).
     * This patch **fails Python style tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `public final class JavaRandomForestExample `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21113669
  
    --- Diff: docs/mllib-gbt.md ---
    @@ -0,0 +1,308 @@
    +---
    +layout: global
    +title: Gradient-Boosted Trees - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Gradient-Boosted Trees
    +---
    +
    +* Table of contents
    +{:toc}
    +
    +[Gradient-Boosted Trees (GBTs)](http://en.wikipedia.org/wiki/Gradient_boosting)
    +are ensembles of [decision trees](mllib-decision-tree.html).
    +GBTs iteratively train decision trees in order to minimize a loss function.
    +Like decision trees, GBTs handle categorical features,
    +extend to the multiclass classification setting, do not require
    +feature scaling, and are able to capture non-linearities and feature interactions.
    +
    +MLlib supports GBTs for binary classification and for regression,
    +using both continuous and categorical features.
    +MLlib implements GBTs using the existing [decision tree](mllib-decision-tree.html) implementation.  Please see the decision tree guide for more information on trees.
    +
    +*Note*: GBTs do not yet support multiclass classification.  For multiclass problems, please use
    +[decision trees](mllib-decision-tree.html) or [Random Forests](mllib-random-forest.html).
    +
    +## Basic algorithm
    +
    +Gradient boosting iteratively trains a sequence of decision trees.
    +On each iteration, the algorithm uses the current ensemble to predict the label of each training instance and then compares the prediction with the true label.  The dataset is re-labeled to put more weight on training instances with poor predictions.  Thus, in the next iteration, the decision tree will help correct for previous mistakes.
    --- End diff --
    
    True, I'll change "weight" to "emphasis."


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21068173
  
    --- Diff: docs/mllib-gbt.md ---
    @@ -0,0 +1,308 @@
    +---
    +layout: global
    +title: Gradient-Boosted Trees - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Gradient-Boosted Trees
    +---
    +
    +* Table of contents
    +{:toc}
    +
    +[Gradient-Boosted Trees (GBTs)](http://en.wikipedia.org/wiki/Gradient_boosting)
    +are ensembles of [decision trees](mllib-decision-tree.html).
    +GBTs iteratively train decision trees in order to minimize a loss function.
    +Like decision trees, GBTs handle categorical features,
    +extend to the multiclass classification setting, do not require
    +feature scaling, and are able to capture non-linearities and feature interactions.
    +
    +MLlib supports GBTs for binary classification and for regression,
    +using both continuous and categorical features.
    +MLlib implements GBTs using the existing [decision tree](mllib-decision-tree.html) implementation.  Please see the decision tree guide for more information on trees.
    +
    +*Note*: GBTs do not yet support multiclass classification.  For multiclass problems, please use
    +[decision trees](mllib-decision-tree.html) or [Random Forests](mllib-random-forest.html).
    +
    +## Basic algorithm
    +
    +Gradient boosting iteratively trains a sequence of decision trees.
    +On each iteration, the algorithm uses the current ensemble to predict the label of each training instance and then compares the prediction with the true label.  The dataset is re-labeled to put more weight on training instances with poor predictions.  Thus, in the next iteration, the decision tree will help correct for previous mistakes.
    +
    +The specific weight mechanism is defined by a loss function (discussed below).  With each iteration, GBTs further reduce this loss function on the training data.
    +
    +### Comparison with Random Forests
    --- End diff --
    
    I really like this section since this is very useful information. We should try and add some graphs here in a separate PR. However, shouldn't this be in a separate section under Ensemble comparing both RF and Boosting algorithms in terms of performance and accuracy.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-65136493
  
      [Test build #23997 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23997/consoleFull) for   PR 3461 at commit [`6fab846`](https://github.com/apache/spark/commit/6fab846ac5c59dd61064bb2965b4bc7cf280f5e5).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `public final class JavaRandomForestExample `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64499904
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23852/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-65136503
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23997/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64510904
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23861/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21111669
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    --- End diff --
    
    will do


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64503905
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23858/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64499893
  
      [Test build #23852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23852/consoleFull) for   PR 3461 at commit [`706d332`](https://github.com/apache/spark/commit/706d332b5e9a0ed85518d5ad62e57a9908169028).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64522984
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23879/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-65352254
  
    @mengxr Sure, that seems like a good solution to the suggestion from @manishamde 
    Will do.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21133672
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    +  * Note that you do not have to specify `categoricalFeaturesInfo`.  The algorithm will still run and may get reasonable results.  However, performance should be better if categorical features are properly designated.
    +
    +### Stopping criteria
    +
    +These parameters determine when the tree stops building (adding new nodes).
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
    +
    +* **`maxDepth`**: Maximum depth of a tree.  Deeper trees are more expressive (potentially allowing higher accuracy), but deeper trees are also more costly to train and are more likely to overfit.
    +
    +* **`minInstancesPerNode`**: For a node to be split further, each of its children must receive at least this number of training instances.  This is commonly used with [Random Forests](mllib-random-forest.html) since those are often trained deeper than individual trees.
    +
    +* **`minInfoGain`**: For a node to be split further, the split must improve at least this much (in terms of information gain).
    +
    +### Tunable parameters
     
    -## Implementation details
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
     
    -### Max memory requirements
    +* **`maxBins`**: Number of bins used when discretizing continuous features.
    +  * Increasing `maxBins` allows the algorithm to consider more split candidates and make fine-grained split decisions.  However, it also increases computation and communication.
    +  * Note that the `maxBins` parameter must be at least the maximum number of categories `$M$` for any categorical feature.
     
    -For faster processing, the decision tree algorithm performs simultaneous histogram computations for
    -all nodes at each level of the tree. This could lead to high memory requirements at deeper levels
    -of the tree, potentially leading to memory overflow errors. To alleviate this problem, a `maxMemoryInMB`
    -training parameter specifies the maximum amount of memory at the workers (twice as much at the
    -master) to be allocated to the histogram computation. The default value is conservatively chosen to
    -be 256 MB to allow the decision algorithm to work in most scenarios. Once the memory requirements
    -for a level-wise computation cross the `maxMemoryInMB` threshold, the node training tasks at each
    -subsequent level are split into smaller tasks.
    +* **`maxMemoryInMB`**: Amount of memory to be used for collecting sufficient statistics.
    +  * The default value is conservatively chosen to be 256 MB to allow the decision algorithm to work in most scenarios.  Increasing `maxMemoryInMB` can lead to faster training (if the memory is available) by allowing fewer passes over the data.  However, there may be decreasing returns as `maxMemoryInMB` grows since the amount of communication on each iteration can be proportional to `maxMemoryInMB`.
    --- End diff --
    
    Ok.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21067802
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    +  * Note that you do not have to specify `categoricalFeaturesInfo`.  The algorithm will still run and may get reasonable results.  However, performance should be better if categorical features are properly designated.
    +
    +### Stopping criteria
    +
    +These parameters determine when the tree stops building (adding new nodes).
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
    --- End diff --
    
    Sentences can be combined.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21111857
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    +  * Note that you do not have to specify `categoricalFeaturesInfo`.  The algorithm will still run and may get reasonable results.  However, performance should be better if categorical features are properly designated.
    +
    +### Stopping criteria
    +
    +These parameters determine when the tree stops building (adding new nodes).
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
    --- End diff --
    
    will modify


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21068117
  
    --- Diff: docs/mllib-gbt.md ---
    @@ -0,0 +1,308 @@
    +---
    +layout: global
    +title: Gradient-Boosted Trees - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Gradient-Boosted Trees
    +---
    +
    +* Table of contents
    +{:toc}
    +
    +[Gradient-Boosted Trees (GBTs)](http://en.wikipedia.org/wiki/Gradient_boosting)
    +are ensembles of [decision trees](mllib-decision-tree.html).
    +GBTs iteratively train decision trees in order to minimize a loss function.
    +Like decision trees, GBTs handle categorical features,
    +extend to the multiclass classification setting, do not require
    +feature scaling, and are able to capture non-linearities and feature interactions.
    +
    +MLlib supports GBTs for binary classification and for regression,
    +using both continuous and categorical features.
    +MLlib implements GBTs using the existing [decision tree](mllib-decision-tree.html) implementation.  Please see the decision tree guide for more information on trees.
    +
    +*Note*: GBTs do not yet support multiclass classification.  For multiclass problems, please use
    +[decision trees](mllib-decision-tree.html) or [Random Forests](mllib-random-forest.html).
    +
    +## Basic algorithm
    +
    +Gradient boosting iteratively trains a sequence of decision trees.
    +On each iteration, the algorithm uses the current ensemble to predict the label of each training instance and then compares the prediction with the true label.  The dataset is re-labeled to put more weight on training instances with poor predictions.  Thus, in the next iteration, the decision tree will help correct for previous mistakes.
    +
    +The specific weight mechanism is defined by a loss function (discussed below).  With each iteration, GBTs further reduce this loss function on the training data.
    +
    +### Comparison with Random Forests
    +
    +Both GBTs and [Random Forests](mllib-random-forest.html) are algorithms for learning ensembles of trees, but the training processes are different.  There are several practical trade-offs:
    +
    + * GBTs may be able to achieve the same accuracy using fewer trees, so the model produced may be smaller (faster for test time prediction).
    + * GBTs train one tree at a time, so they can take longer to train than random forests.  Random Forests can train multiple trees in parallel.
    +   * On the other hand, it is often reasonable to use smaller trees with GBTs than with Random Forests, and training smaller trees takes less time.
    + * Random Forests can be less prone to overfitting.  Training more trees in a Random Forest reduces the likelihood of overfitting, but training more trees with GBTs increases the likelihood of overfitting.
    +
    +In short, both algorithms can be effective.  GBTs may be more useful if test time prediction speed is important.  Random Forests are arguably more successful in industry.
    --- End diff --
    
    I think we should avoid this lest people start quoting us in the future. :-) There is a lot of results comparing RF and Boosting and the results are mixed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-65322960
  
    @jkbradley minor: Shall we merge RF and GBT into a single section called "tree ensembles (random forests and gradient-boosted trees" (on the same level as decision trees) ? Then we can move the comparison part to the bottom (or to the very beginning).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21113959
  
    --- Diff: docs/mllib-gbt.md ---
    @@ -0,0 +1,308 @@
    +---
    +layout: global
    +title: Gradient-Boosted Trees - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Gradient-Boosted Trees
    +---
    +
    +* Table of contents
    +{:toc}
    +
    +[Gradient-Boosted Trees (GBTs)](http://en.wikipedia.org/wiki/Gradient_boosting)
    +are ensembles of [decision trees](mllib-decision-tree.html).
    +GBTs iteratively train decision trees in order to minimize a loss function.
    +Like decision trees, GBTs handle categorical features,
    +extend to the multiclass classification setting, do not require
    +feature scaling, and are able to capture non-linearities and feature interactions.
    +
    +MLlib supports GBTs for binary classification and for regression,
    +using both continuous and categorical features.
    +MLlib implements GBTs using the existing [decision tree](mllib-decision-tree.html) implementation.  Please see the decision tree guide for more information on trees.
    +
    +*Note*: GBTs do not yet support multiclass classification.  For multiclass problems, please use
    +[decision trees](mllib-decision-tree.html) or [Random Forests](mllib-random-forest.html).
    +
    +## Basic algorithm
    +
    +Gradient boosting iteratively trains a sequence of decision trees.
    +On each iteration, the algorithm uses the current ensemble to predict the label of each training instance and then compares the prediction with the true label.  The dataset is re-labeled to put more weight on training instances with poor predictions.  Thus, in the next iteration, the decision tree will help correct for previous mistakes.
    +
    +The specific weight mechanism is defined by a loss function (discussed below).  With each iteration, GBTs further reduce this loss function on the training data.
    +
    +### Comparison with Random Forests
    +
    +Both GBTs and [Random Forests](mllib-random-forest.html) are algorithms for learning ensembles of trees, but the training processes are different.  There are several practical trade-offs:
    +
    + * GBTs may be able to achieve the same accuracy using fewer trees, so the model produced may be smaller (faster for test time prediction).
    + * GBTs train one tree at a time, so they can take longer to train than random forests.  Random Forests can train multiple trees in parallel.
    +   * On the other hand, it is often reasonable to use smaller trees with GBTs than with Random Forests, and training smaller trees takes less time.
    + * Random Forests can be less prone to overfitting.  Training more trees in a Random Forest reduces the likelihood of overfitting, but training more trees with GBTs increases the likelihood of overfitting.
    +
    +In short, both algorithms can be effective.  GBTs may be more useful if test time prediction speed is important.  Random Forests are arguably more successful in industry.
    --- End diff --
    
    I'll say less : )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64522980
  
      [Test build #23879 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23879/consoleFull) for   PR 3461 at commit [`b9f8576`](https://github.com/apache/spark/commit/b9f8576100878a2bccae6179ded6c50721722641).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `public final class JavaRandomForestExample `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21067979
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    +  * Note that you do not have to specify `categoricalFeaturesInfo`.  The algorithm will still run and may get reasonable results.  However, performance should be better if categorical features are properly designated.
    +
    +### Stopping criteria
    +
    +These parameters determine when the tree stops building (adding new nodes).
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
    +
    +* **`maxDepth`**: Maximum depth of a tree.  Deeper trees are more expressive (potentially allowing higher accuracy), but deeper trees are also more costly to train and are more likely to overfit.
    +
    +* **`minInstancesPerNode`**: For a node to be split further, each of its children must receive at least this number of training instances.  This is commonly used with [Random Forests](mllib-random-forest.html) since those are often trained deeper than individual trees.
    +
    +* **`minInfoGain`**: For a node to be split further, the split must improve at least this much (in terms of information gain).
    +
    +### Tunable parameters
     
    -## Implementation details
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
     
    -### Max memory requirements
    +* **`maxBins`**: Number of bins used when discretizing continuous features.
    +  * Increasing `maxBins` allows the algorithm to consider more split candidates and make fine-grained split decisions.  However, it also increases computation and communication.
    +  * Note that the `maxBins` parameter must be at least the maximum number of categories `$M$` for any categorical feature.
     
    -For faster processing, the decision tree algorithm performs simultaneous histogram computations for
    -all nodes at each level of the tree. This could lead to high memory requirements at deeper levels
    -of the tree, potentially leading to memory overflow errors. To alleviate this problem, a `maxMemoryInMB`
    -training parameter specifies the maximum amount of memory at the workers (twice as much at the
    -master) to be allocated to the histogram computation. The default value is conservatively chosen to
    -be 256 MB to allow the decision algorithm to work in most scenarios. Once the memory requirements
    -for a level-wise computation cross the `maxMemoryInMB` threshold, the node training tasks at each
    -subsequent level are split into smaller tasks.
    +* **`maxMemoryInMB`**: Amount of memory to be used for collecting sufficient statistics.
    +  * The default value is conservatively chosen to be 256 MB to allow the decision algorithm to work in most scenarios.  Increasing `maxMemoryInMB` can lead to faster training (if the memory is available) by allowing fewer passes over the data.  However, there may be decreasing returns as `maxMemoryInMB` grows since the amount of communication on each iteration can be proportional to `maxMemoryInMB`.
    --- End diff --
    
    ```However, there may be decreasing returns as `maxMemoryInMB` grows since the amount of communication on each iteration can be proportional to `maxMemoryInMB`.```
    Not sure. Aren't we communicating the same amount of data per level/iteration irrespective of the batch size?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21067860
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    +  * Note that you do not have to specify `categoricalFeaturesInfo`.  The algorithm will still run and may get reasonable results.  However, performance should be better if categorical features are properly designated.
    +
    +### Stopping criteria
    --- End diff --
    
    We could mention the default values for all optional parameters here.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64517610
  
      [Test build #23879 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23879/consoleFull) for   PR 3461 at commit [`b9f8576`](https://github.com/apache/spark/commit/b9f8576100878a2bccae6179ded6c50721722641).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21067826
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    +  * Note that you do not have to specify `categoricalFeaturesInfo`.  The algorithm will still run and may get reasonable results.  However, performance should be better if categorical features are properly designated.
    +
    +### Stopping criteria
    +
    +These parameters determine when the tree stops building (adding new nodes).
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
    +
    +* **`maxDepth`**: Maximum depth of a tree.  Deeper trees are more expressive (potentially allowing higher accuracy), but deeper trees are also more costly to train and are more likely to overfit.
    --- End diff --
    
    Nit: avoid using deeper trees twice in a sentence.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21114104
  
    --- Diff: docs/mllib-gbt.md ---
    @@ -0,0 +1,308 @@
    +---
    +layout: global
    +title: Gradient-Boosted Trees - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Gradient-Boosted Trees
    +---
    +
    +* Table of contents
    +{:toc}
    +
    +[Gradient-Boosted Trees (GBTs)](http://en.wikipedia.org/wiki/Gradient_boosting)
    +are ensembles of [decision trees](mllib-decision-tree.html).
    +GBTs iteratively train decision trees in order to minimize a loss function.
    +Like decision trees, GBTs handle categorical features,
    +extend to the multiclass classification setting, do not require
    +feature scaling, and are able to capture non-linearities and feature interactions.
    +
    +MLlib supports GBTs for binary classification and for regression,
    +using both continuous and categorical features.
    +MLlib implements GBTs using the existing [decision tree](mllib-decision-tree.html) implementation.  Please see the decision tree guide for more information on trees.
    +
    +*Note*: GBTs do not yet support multiclass classification.  For multiclass problems, please use
    +[decision trees](mllib-decision-tree.html) or [Random Forests](mllib-random-forest.html).
    +
    +## Basic algorithm
    +
    +Gradient boosting iteratively trains a sequence of decision trees.
    +On each iteration, the algorithm uses the current ensemble to predict the label of each training instance and then compares the prediction with the true label.  The dataset is re-labeled to put more weight on training instances with poor predictions.  Thus, in the next iteration, the decision tree will help correct for previous mistakes.
    +
    +The specific weight mechanism is defined by a loss function (discussed below).  With each iteration, GBTs further reduce this loss function on the training data.
    +
    +### Comparison with Random Forests
    --- End diff --
    
    Should we have a new Ensembles section in the guide?  It might be pretty short.  (And I haven't seen experimental results in the guide; would they belong elsewhere?)
    
    Eventually, I could imagine either (a) an Ensembles section once we have more ensemble algs or (b) a section in the guide covering all algorithms and how to choose between them.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64518822
  
    OK!  I think everything's updated, though I'm sure people will have feedback.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21112016
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    +  * Note that you do not have to specify `categoricalFeaturesInfo`.  The algorithm will still run and may get reasonable results.  However, performance should be better if categorical features are properly designated.
    +
    +### Stopping criteria
    +
    +These parameters determine when the tree stops building (adding new nodes).
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
    +
    +* **`maxDepth`**: Maximum depth of a tree.  Deeper trees are more expressive (potentially allowing higher accuracy), but deeper trees are also more costly to train and are more likely to overfit.
    --- End diff --
    
    will do


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21113406
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    +  * Note that you do not have to specify `categoricalFeaturesInfo`.  The algorithm will still run and may get reasonable results.  However, performance should be better if categorical features are properly designated.
    +
    +### Stopping criteria
    +
    +These parameters determine when the tree stops building (adding new nodes).
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
    +
    +* **`maxDepth`**: Maximum depth of a tree.  Deeper trees are more expressive (potentially allowing higher accuracy), but deeper trees are also more costly to train and are more likely to overfit.
    +
    +* **`minInstancesPerNode`**: For a node to be split further, each of its children must receive at least this number of training instances.  This is commonly used with [Random Forests](mllib-random-forest.html) since those are often trained deeper than individual trees.
    +
    +* **`minInfoGain`**: For a node to be split further, the split must improve at least this much (in terms of information gain).
    +
    +### Tunable parameters
     
    -## Implementation details
    +These parameters may be tuned.  Be careful to validate on held-out test data when tuning in order to avoid overfitting.
     
    -### Max memory requirements
    +* **`maxBins`**: Number of bins used when discretizing continuous features.
    +  * Increasing `maxBins` allows the algorithm to consider more split candidates and make fine-grained split decisions.  However, it also increases computation and communication.
    +  * Note that the `maxBins` parameter must be at least the maximum number of categories `$M$` for any categorical feature.
     
    -For faster processing, the decision tree algorithm performs simultaneous histogram computations for
    -all nodes at each level of the tree. This could lead to high memory requirements at deeper levels
    -of the tree, potentially leading to memory overflow errors. To alleviate this problem, a `maxMemoryInMB`
    -training parameter specifies the maximum amount of memory at the workers (twice as much at the
    -master) to be allocated to the histogram computation. The default value is conservatively chosen to
    -be 256 MB to allow the decision algorithm to work in most scenarios. Once the memory requirements
    -for a level-wise computation cross the `maxMemoryInMB` threshold, the node training tasks at each
    -subsequent level are split into smaller tasks.
    +* **`maxMemoryInMB`**: Amount of memory to be used for collecting sufficient statistics.
    +  * The default value is conservatively chosen to be 256 MB to allow the decision algorithm to work in most scenarios.  Increasing `maxMemoryInMB` can lead to faster training (if the memory is available) by allowing fewer passes over the data.  However, there may be decreasing returns as `maxMemoryInMB` grows since the amount of communication on each iteration can be proportional to `maxMemoryInMB`.
    --- End diff --
    
    We communicate a fixed amount of data for every node in the tree which we eventually train.  Increasing maxMemoryInMB will increase the number of nodes we train in each iteration, but it doesn't change the overall amount of communication required (ignoring overhead for very small maxMemoryInMB).  Once we're training a fair number of nodes in each iteration, the overhead (making a pass over the data + latency in communication) becomes small relative to other costs (computing stats + communicating stats).  Once the overhead is small, then training more nodes per iteration doesn't help much.  This is what I've observed in tests, at least.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21112912
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    +  * Note that you do not have to specify `categoricalFeaturesInfo`.  The algorithm will still run and may get reasonable results.  However, performance should be better if categorical features are properly designated.
    +
    +### Stopping criteria
    --- End diff --
    
    I have mixed feelings about that.  Do default parameters belong here, or just in the API docs?  (It's easy for these docs to get out of date if defaults change.)  @mengxr @codedeft If you have thoughts, let me know.
    I'd be OK either way.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64499901
  
      [Test build #23852 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23852/consoleFull) for   PR 3461 at commit [`706d332`](https://github.com/apache/spark/commit/706d332b5e9a0ed85518d5ad62e57a9908169028).
     * This patch **fails RAT tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `public final class JavaRandomForestClassification `
      * `public final class JavaRandomForestRegression `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64503813
  
      [Test build #23858 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23858/consoleFull) for   PR 3461 at commit [`2b60b6e`](https://github.com/apache/spark/commit/2b60b6efd9284b9cbd7a80d1abbffc9e95d2ff0d).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21067657
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    --- End diff --
    
    Rephrase? The parameters are listed here in descending order of importance.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by manishamde <gi...@git.apache.org>.
Github user manishamde commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21068084
  
    --- Diff: docs/mllib-gbt.md ---
    @@ -0,0 +1,308 @@
    +---
    +layout: global
    +title: Gradient-Boosted Trees - MLlib
    +displayTitle: <a href="mllib-guide.html">MLlib</a> - Gradient-Boosted Trees
    +---
    +
    +* Table of contents
    +{:toc}
    +
    +[Gradient-Boosted Trees (GBTs)](http://en.wikipedia.org/wiki/Gradient_boosting)
    +are ensembles of [decision trees](mllib-decision-tree.html).
    +GBTs iteratively train decision trees in order to minimize a loss function.
    +Like decision trees, GBTs handle categorical features,
    +extend to the multiclass classification setting, do not require
    +feature scaling, and are able to capture non-linearities and feature interactions.
    +
    +MLlib supports GBTs for binary classification and for regression,
    +using both continuous and categorical features.
    +MLlib implements GBTs using the existing [decision tree](mllib-decision-tree.html) implementation.  Please see the decision tree guide for more information on trees.
    +
    +*Note*: GBTs do not yet support multiclass classification.  For multiclass problems, please use
    +[decision trees](mllib-decision-tree.html) or [Random Forests](mllib-random-forest.html).
    +
    +## Basic algorithm
    +
    +Gradient boosting iteratively trains a sequence of decision trees.
    +On each iteration, the algorithm uses the current ensemble to predict the label of each training instance and then compares the prediction with the true label.  The dataset is re-labeled to put more weight on training instances with poor predictions.  Thus, in the next iteration, the decision tree will help correct for previous mistakes.
    --- End diff --
    
    The weight-based relabeling sounds more like AdaBoost.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-65125044
  
      [Test build #23997 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23997/consoleFull) for   PR 3461 at commit [`6fab846`](https://github.com/apache/spark/commit/6fab846ac5c59dd61064bb2965b4bc7cf280f5e5).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64506845
  
    Note: I'm working on updating the decision tree programming guide further too (with more info about parameters).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-65124916
  
    @manishamde Thanks for the feedback!  I made the fixes, except for the default values for all optional parameters + ensembles section issues.  Let me know if you disagree strongly, and we can see if others have thoughts too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64510902
  
      [Test build #23861 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23861/consoleFull) for   PR 3461 at commit [`375204c`](https://github.com/apache/spark/commit/375204ce0c6a3e360d4b8e1b61214552ac344ecc).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds the following public classes _(experimental)_:
      * `public final class JavaRandomForestExample `



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/3461#issuecomment-64504928
  
      [Test build #23861 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23861/consoleFull) for   PR 3461 at commit [`375204c`](https://github.com/apache/spark/commit/375204ce0c6a3e360d4b8e1b61214552ac344ecc).
     * This patch merges cleanly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-4580] [SPARK-4610] [mllib] Documentatio...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:

    https://github.com/apache/spark/pull/3461#discussion_r21111725
  
    --- Diff: docs/mllib-decision-tree.md ---
    @@ -103,36 +106,73 @@ and the resulting `$M-1$` split candidates are considered.
     
     ### Stopping rule
     
    -The recursive tree construction is stopped at a node when one of the two conditions is met:
    +The recursive tree construction is stopped at a node when one of the following conditions is met:
     
     1. The node depth is equal to the `maxDepth` training parameter.
    -2. No split candidate leads to an information gain at the node.
    +2. No split candidate leads to an information gain greater than `minInfoGain`.
    +3. No split candidate produces child nodes which each have at least `minInstancesPerNode` training instances.
    +
    +## Usage tips
    +
    +We include a few guidelines for using decision trees by discussing the various parameters.
    +There are many parameters, put in order here with the most imporant first.  New users should mainly consider the "Problem specification parameters" section below and the `maxDepth` parameter.
    +
    +### Problem specification parameters
    +
    +These parameters describe the problem you want to solve and your dataset.
    +They should be specified and do not require tuning.
    +
    +* **`algo`**: `Classification` or `Regression`
    +
    +* **`numClasses`**: Number of classes (for `Classification` only)
    +
    +* **`categoricalFeaturesInfo`**: Specifies which features are categorical and how many categorical values each of those features can take.  This is given as a map from feature indices to feature arity (number of categories).  Any features not in this map are treated as continuous.
    +  * E.g., `Map(1 -> 2, 4 -> 10)` specifies that feature `1` is binary (taking values `0` or `1`) and that feature `4` has 10 categories (values `{0, 1, ..., 9}`).  Note that feature indices are 0-based: features `1` and `4` are the 2nd and 5th elements of an instance's feature vector.
    --- End diff --
    
    good idea


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org