You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yongtang <gi...@git.apache.org> on 2016/04/13 17:08:26 UTC

[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

GitHub user yongtang opened a pull request:

    https://github.com/apache/spark/pull/12360

    [SPARK-14565][ML] RandomForest should use parseInt and parseDouble for feature subset size instead of regexes

    ## What changes were proposed in this pull request?
    
    This fix tries to change RandomForest's supported strategies from using regexes to using parseInt and
    parseDouble, for the purpose of robustness and maintainability.
    
    ## How was this patch tested?
    
    Existing tests passed.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yongtang/spark SPARK-14565

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/12360.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #12360
    
----
commit 57456d352ae9b45596ac776341ac09199aadc30c
Author: Yong Tang <yo...@outlook.com>
Date:   2016-04-13T15:04:55Z

    [SPARK-14565][ML] RandomForest should use parseInt and parseDouble for feature subset size instead of regexes
    
    This fix tries to change RandomForest's supported strategies from using regexes to using parseInt and
    parseDouble, for the purpose of robustness and maintainability.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209770009
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209549793
  
    **[Test build #55726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55726/consoleFull)** for PR 12360 at commit [`57456d3`](https://github.com/apache/spark/commit/57456d352ae9b45596ac776341ac09199aadc30c).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209677122
  
    One more comment: I'd like to make sure there is always at least 1 feature being used.  Could you please update the parsing and add that to the unit test in RandomForestSuite.scala?  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209549959
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55726/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209769942
  
    **[Test build #55794 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55794/consoleFull)** for PR 12360 at commit [`ed346cd`](https://github.com/apache/spark/commit/ed346cd0f8b07876f95ae9636f7714e6c5f26c13).
     * This patch passes all tests.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12360#discussion_r59580909
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala ---
    @@ -346,10 +346,12 @@ private[ml] trait HasFeatureSubsetStrategy extends Params {
        */
       final val featureSubsetStrategy: Param[String] = new Param[String](this, "featureSubsetStrategy",
         "The number of features to consider for splits at each tree node." +
    -      s" Supported options: ${RandomForestParams.supportedFeatureSubsetStrategies.mkString(", ")}",
    +      s" Supported options: ${RandomForestParams.supportedFeatureSubsetStrategies.mkString(", ")}" +
    +      s", (0.0-1.0], [1-n].",
         (value: String) =>
           RandomForestParams.supportedFeatureSubsetStrategies.contains(value.toLowerCase)
    -      || value.matches(RandomForestParams.supportedFeatureSubsetStrategiesRegex))
    +      || RandomForestParams.integerFeatureSubsetStrategy.unapply(value).isDefined
    --- End diff --
    
    `Try(value.toDouble).filter(_ > 0.0).isSuccess`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-210218020
  
    Merged into master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209535208
  
    **[Test build #55726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55726/consoleFull)** for PR 12360 at commit [`57456d3`](https://github.com/apache/spark/commit/57456d352ae9b45596ac776341ac09199aadc30c).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209770010
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/55794/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209549957
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209760563
  
    **[Test build #55794 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/55794/consoleFull)** for PR 12360 at commit [`ed346cd`](https://github.com/apache/spark/commit/ed346cd0f8b07876f95ae9636f7714e6c5f26c13).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209534593
  
    ok to test


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209500271
  
    Can one of the admins verify this patch?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12360#discussion_r59580903
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/impl/DecisionTreeMetadata.scala ---
    @@ -184,15 +185,15 @@ private[spark] object DecisionTreeMetadata extends Logging {
           case _ => featureSubsetStrategy
         }
     
    -    val isIntRegex = "^([1-9]\\d*)$".r
    -    val isFractionRegex = "^(0?\\.\\d*[1-9]\\d*|1\\.0+)$".r
         val numFeaturesPerNode: Int = _featureSubsetStrategy match {
           case "all" => numFeatures
           case "sqrt" => math.sqrt(numFeatures).ceil.toInt
           case "log2" => math.max(1, (math.log(numFeatures) / math.log(2)).ceil.toInt)
           case "onethird" => (numFeatures / 3.0).ceil.toInt
    -      case isIntRegex(number) => if (BigInt(number) > numFeatures) numFeatures else number.toInt
    -      case isFractionRegex(fraction) => (fraction.toDouble * numFeatures).ceil.toInt
    +      case RandomForestParams.integerFeatureSubsetStrategy(number) =>
    --- End diff --
    
    I think we should simplify the checks.
    
    ~~~scala
    import scala.util.Try
    
    val numFeaturesPerNode: Int = _featureSubsetStrategy match {
       ...
       case _ =>
         Try(_featureSubsetStrategy.toDouble).toOption match {
           case Some(value) =>
             if (value <= 0.0) {
               throw new IllegalArgumentException(...)
             } else if (value < 1.0) {
               math.ceil(value * numFeatures)
             } else {
               math.ceil(value)
             }
           case None =>
             throw new IllegalArgumentException(...)
         }
    }
    ~~~
    
    See my comment below for the validator.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209763023
  
    LGTM pending Jenkins


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by yongtang <gi...@git.apache.org>.
Github user yongtang commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209759774
  
    Thanks @srowen @mengxr @jkbradley. The pull request has been updated. Please let me know if there are any further issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/12360


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209643510
  
    @mengxr I'd prefer to treat "1" as numFeaturesPerNode = 1 and "1.0" as equivalent to "all" in order to match sklearn's semantics.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on a diff in the pull request:

    https://github.com/apache/spark/pull/12360#discussion_r59567223
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/treeParams.scala ---
    @@ -397,8 +399,31 @@ private[spark] object RandomForestParams {
       final val supportedFeatureSubsetStrategies: Array[String] =
         Array("auto", "all", "onethird", "sqrt", "log2").map(_.toLowerCase)
     
    -  // The regex to capture "(0.0-1.0]", and "n" for integer 0 < n <= (number of features)
    -  final val supportedFeatureSubsetStrategiesRegex = "^(?:[1-9]\\d*|0?\\.\\d*[1-9]\\d*|1\\.0+)$"
    +  object integerFeatureSubsetStrategy {
    +    def unapply(strategy: String): Option[Int] = try {
    +      val number = strategy.toInt
    +      if (0 < number) {
    --- End diff --
    
    Nit: number > 0? and import java.lang.NumberFormatException


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-14565][ML] RandomForest should use pars...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/12360#issuecomment-209535636
  
    Besides inline comments, we should also consider the behavior at `1.0`. Since `all` is already an option, maybe we should treat `1.0` as `numFeaturesPerNode = 1` instead of `all`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org