You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yanboliang <gi...@git.apache.org> on 2015/09/19 08:55:53 UTC

[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

GitHub user yanboliang opened a pull request:

    https://github.com/apache/spark/pull/8836

    [SPARK-10686] [ML] Add quantilesCol to AFTSurvivalRegression

    By default ```quantilesCol``` should be empty. If ```quantileProbabilities``` is set, we should append quantiles as a new column (of type Vector).

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/yanboliang/spark spark-10686

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/spark/pull/8836.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #8836
    
----
commit 1d8a6828cc7078b6cba229148e9da923f8507848
Author: Yanbo Liang <yb...@gmail.com>
Date:   2015-09-19T06:53:43Z

    Add quantilesCol to AFTSurvivalRegression

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r39995446
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
    @@ -71,11 +71,29 @@ private[regression] trait AFTSurvivalRegressionParams extends Params
       @Since("1.6.0")
       def getQuantileProbabilities: Array[Double] = $(quantileProbabilities)
     
    +  /**
    +   * Param for quantiles column name.
    +   * If quantileProbabilities is set, quantiles of corresponding probabilities
    +   * will be outputed in this column.
    +   * @group param
    +   */
    +  @Since("1.6.0")
    +  final val quantilesCol: Param[String] = new Param(this, "quantilesCol", "quantiles column name")
    --- End diff --
    
    Set default value to `""`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141844411
  
    The changes look good to me. Just want to discuss the semantics to enable quantiles in output. We set a default value for `quantilesCol`, and use `quantileProbabilities` to control whether to output quantiles. Another issue is that `predictQuantiles` requires `quantileProbabilities`. It means in order to use `predictQuantiles`, we need to append a new column in `transform`. It is not straightforward to understand the logic here.
    
    I would suggest making `quantilesCol` default to an empty string and ask users to explicitly set both `quantilesCol` and `quantileProbabilities` to enable quantile output.
    
    + @rotationsymmetry 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40125900
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala ---
    @@ -167,10 +169,18 @@ class AFTSurvivalRegressionSuite extends SparkFunSuite with MLlibTestSparkContex
         model.setQuantileProbabilities(quantileProbabilities)
         assert(model.predictQuantiles(features) ~== quantilePredictR relTol 1E-3)
     
    -    model.transform(datasetUnivariate).select("features", "prediction").collect().foreach {
    -      case Row(features: DenseVector, prediction1: Double) =>
    -        val prediction2 = math.exp(BLAS.dot(model.coefficients, features) + model.intercept)
    +    model.setQuantilesCol("quantiles")
    --- End diff --
    
    This could go to estimator as well, unless we want to test changing `quantilesCol` after fitting.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40125908
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala ---
    @@ -167,10 +169,18 @@ class AFTSurvivalRegressionSuite extends SparkFunSuite with MLlibTestSparkContex
         model.setQuantileProbabilities(quantileProbabilities)
         assert(model.predictQuantiles(features) ~== quantilePredictR relTol 1E-3)
     
    -    model.transform(datasetUnivariate).select("features", "prediction").collect().foreach {
    -      case Row(features: DenseVector, prediction1: Double) =>
    -        val prediction2 = math.exp(BLAS.dot(model.coefficients, features) + model.intercept)
    +    model.setQuantilesCol("quantiles")
    +    model.transform(datasetUnivariate).select("features", "prediction", "quantiles")
    +      .collect().foreach {
    +      case Row(features: DenseVector, prediction1: Double, predictionQuantiles1: Vector) =>
    +        val lambda = math.exp(BLAS.dot(model.coefficients, features) + model.intercept)
    --- End diff --
    
    I'm not sure whether this block is useful or not. We already verified that `model.predict` and `model.predictQuantiles` work correctly. So here we only need to verify the output values are indeed `prediction` and `quantiles`. So we could use `model.predict` and `model.predictQuantiles` here::
    
    ~~~scala
    model.transform(datasetUnivariate).select("features", "prediction", "quantiles")
      .collect().foreach {
        case Row(features: Vector, prediction: Double, quantiles: Vector) =>
          assert(prediction ~== model.predict(features) relTol 1E-5)
          assert(quantiles ~== model.predictQuantiles(features) relTol 1E-5)
      }
    ~~~
    
    Also note the indentation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142553914
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42901/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142532551
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141695982
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42711/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141691815
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142553910
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142183841
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:

    https://github.com/apache/spark/pull/8836


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40124054
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
    @@ -90,6 +108,12 @@ private[regression] trait AFTSurvivalRegressionParams extends Params
           SchemaUtils.checkColumnType(schema, $(censorCol), DoubleType)
           SchemaUtils.checkColumnType(schema, $(labelCol), DoubleType)
         }
    +    if (hasQuantileProbabilities && hasQuantilesCol) {
    +      SchemaUtils.appendColumn(schema, $(quantilesCol), new VectorUDT)
    +    } else if (hasQuantileProbabilities || hasQuantilesCol) {
    +      logWarning("The output will not include quantiles column. " +
    +        "If you want to include, please set both quantileProbabilities and quantilesCol.")
    +    }
    --- End diff --
    
    I think if users set `quantilesCol` but not `quantileProbabilities`, we should throw an exception rather than an warning message because users should expect `quantilesCol` in the output. Actually, I think it is more convenient to set the default value of `quantileProbabilities` to `[0.25, 0.5, 0.75]`, and then require a non-empty array for this param.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40125943
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala ---
    @@ -302,7 +320,44 @@ class AFTSurvivalRegressionSuite extends SparkFunSuite with MLlibTestSparkContex
         model.setQuantileProbabilities(quantileProbabilities)
         assert(model.predictQuantiles(features) ~== quantilePredictR relTol 1E-3)
     
    -    model.transform(datasetMultivariate).select("features", "prediction").collect().foreach {
    +    model.setQuantilesCol("quantiles")
    +    model.transform(datasetMultivariate).select("features", "prediction", "quantiles")
    +      .collect().foreach {
    +      case Row(features: DenseVector, prediction1: Double, predictionQuantiles1: Vector) =>
    +        val lambda = math.exp(BLAS.dot(model.coefficients, features) + model.intercept)
    --- End diff --
    
    Ditto.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141949117
  
      [Test build #42750 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42750/console) for   PR 8836 at commit [`6b86903`](https://github.com/apache/spark/commit/6b8690307275fecd4672df40ea576a3e5083517a).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141941970
  
      [Test build #42750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42750/consoleFull) for   PR 8836 at commit [`6b86903`](https://github.com/apache/spark/commit/6b8690307275fecd4672df40ea576a3e5083517a).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141695814
  
      [Test build #42711 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42711/console) for   PR 8836 at commit [`1d8a682`](https://github.com/apache/spark/commit/1d8a6828cc7078b6cba229148e9da923f8507848).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142179260
  
    Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142525175
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by rotationsymmetry <gi...@git.apache.org>.
Github user rotationsymmetry commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40056721
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
    @@ -90,6 +108,12 @@ private[regression] trait AFTSurvivalRegressionParams extends Params
           SchemaUtils.checkColumnType(schema, $(censorCol), DoubleType)
           SchemaUtils.checkColumnType(schema, $(labelCol), DoubleType)
         }
    +    if (hasQuantileProbabilities && hasQuantilesCol) {
    +      SchemaUtils.appendColumn(schema, $(quantilesCol), new VectorUDT)
    +    } else if (hasQuantileProbabilities || hasQuantilesCol) {
    +      logWarning("The output will not include quantiles column. " +
    +        "If you want to include, please set both quantileProbabilities and quantilesCol.")
    +    }
    --- End diff --
    
    I think the sematic is clear to me. On the other hand, can we consider these two as a combo, i.e. 
    case class QuantileParams(quantileProbabilities: Array[Double], quantilesCol: String)
    Then the user will have to set both items simultaneously.
    
    Not sure it will work but try to brain storm. :)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141949196
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42750/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142178996
  
    Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142176067
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142525540
  
      [Test build #42897 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42897/consoleFull) for   PR 8836 at commit [`417c8cb`](https://github.com/apache/spark/commit/417c8cb5aabb23cf80f40d83bb66832e354eae16).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142178998
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42810/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142532515
  
      [Test build #42897 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42897/console) for   PR 8836 at commit [`417c8cb`](https://github.com/apache/spark/commit/417c8cb5aabb23cf80f40d83bb66832e354eae16).
     * This patch **fails Spark unit tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142532552
  
    Test FAILed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42897/
    Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142176025
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40059910
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
    @@ -90,6 +108,12 @@ private[regression] trait AFTSurvivalRegressionParams extends Params
           SchemaUtils.checkColumnType(schema, $(censorCol), DoubleType)
           SchemaUtils.checkColumnType(schema, $(labelCol), DoubleType)
         }
    +    if (hasQuantileProbabilities && hasQuantilesCol) {
    +      SchemaUtils.appendColumn(schema, $(quantilesCol), new VectorUDT)
    +    } else if (hasQuantileProbabilities || hasQuantilesCol) {
    +      logWarning("The output will not include quantiles column. " +
    +        "If you want to include, please set both quantileProbabilities and quantilesCol.")
    +    }
    --- End diff --
    
    @rotationsymmetry Thanks for your comments! I think we should keep the same tradition with R which have a default prediction and some other predictions cloud be enabled by parameter.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142179770
  
      [Test build #42813 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42813/consoleFull) for   PR 8836 at commit [`34c518d`](https://github.com/apache/spark/commit/34c518d5705458889f8c06f748bcb945c3f23519).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142553692
  
      [Test build #42901 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42901/console) for   PR 8836 at commit [`417c8cb`](https://github.com/apache/spark/commit/417c8cb5aabb23cf80f40d83bb66832e354eae16).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141949192
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141691804
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40125898
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala ---
    @@ -60,12 +60,14 @@ class AFTSurvivalRegressionSuite extends SparkFunSuite with MLlibTestSparkContex
         assert(aftr.getMaxIter === 100)
         assert(aftr.getTol === 1E-6)
         val model = aftr.fit(datasetUnivariate)
    +      .setQuantileProbabilities(Array(0.1, 0.8))
    --- End diff --
    
    It is better to put setters before fit to make the estimator and the model it produces consistent.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141940894
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142543868
  
      [Test build #42901 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42901/consoleFull) for   PR 8836 at commit [`417c8cb`](https://github.com/apache/spark/commit/417c8cb5aabb23cf80f40d83bb66832e354eae16).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142539325
  
    Jenkins, test this please.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142179290
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141940872
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142183786
  
      [Test build #42813 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42813/console) for   PR 8836 at commit [`34c518d`](https://github.com/apache/spark/commit/34c518d5705458889f8c06f748bcb945c3f23519).
     * This patch **passes all tests**.
     * This patch merges cleanly.
     * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40051968
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
    @@ -90,6 +108,12 @@ private[regression] trait AFTSurvivalRegressionParams extends Params
           SchemaUtils.checkColumnType(schema, $(censorCol), DoubleType)
           SchemaUtils.checkColumnType(schema, $(labelCol), DoubleType)
         }
    +    if (hasQuantileProbabilities && hasQuantilesCol) {
    +      SchemaUtils.appendColumn(schema, $(quantilesCol), new VectorUDT)
    +    } else if (hasQuantileProbabilities || hasQuantilesCol) {
    +      logWarning("The output will not include quantiles column. " +
    +        "If you want to include, please set both quantileProbabilities and quantilesCol.")
    +    }
    --- End diff --
    
    The semantic:
    1, Users set both ```quantileProbabilities``` and ```quantilesCol``` => output w/ quantiles column
    2, Users set ```quantileProbabilities``` but not ```quantilesCol``` => print warning here, output w/o quantiles column and users can use ```predictQuantiles```
    3, Users set ```quantilesCol``` but not ```quantileProbabilities``` => print warning here and throw ```IllegalArgumentException``` at [L284](https://github.com/yanboliang/spark/blob/spark-10686/mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala#L284)
    4, Neither of ```quantileProbabilities``` and ```quantilesCol``` is set =>  Users purpose to output w/o quantiles column


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142043877
  
    @yanboliang Shall we have a test without "quantilesCol" or having "quantilesProbabilities` but not `quantilesCol`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142540830
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40126117
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala ---
    @@ -302,7 +320,44 @@ class AFTSurvivalRegressionSuite extends SparkFunSuite with MLlibTestSparkContex
         model.setQuantileProbabilities(quantileProbabilities)
         assert(model.predictQuantiles(features) ~== quantilePredictR relTol 1E-3)
     
    -    model.transform(datasetMultivariate).select("features", "prediction").collect().foreach {
    +    model.setQuantilesCol("quantiles")
    +    model.transform(datasetMultivariate).select("features", "prediction", "quantiles")
    +      .collect().foreach {
    +      case Row(features: DenseVector, prediction1: Double, predictionQuantiles1: Vector) =>
    +        val lambda = math.exp(BLAS.dot(model.coefficients, features) + model.intercept)
    +        val k = 1 / model.scale
    +        val prediction2 = lambda
    +        val predictionQuantiles2 = Vectors.dense(model.getQuantileProbabilities.map {
    +          q => lambda * math.exp(math.log(-math.log(1 - q)) / k)
    +        })
    +        assert(prediction1 ~== prediction2 relTol 1E-5)
    +        assert(predictionQuantiles1 ~== predictionQuantiles2 relTol 1E-5)
    +    }
    +  }
    +
    +  test("aft survival regression w/o quantiles column") {
    +    val trainer = new AFTSurvivalRegression
    +
    +    val features = Vectors.dense(6.559282795753792)
    +    val quantileProbabilities = Array(0.1, 0.5, 0.9)
    +    val quantilePredict = Vectors.dense(0.1879174, 2.6801195, 14.5779394)
    +
    +    val model = trainer.fit(datasetUnivariate)
    +
    +    intercept[IllegalArgumentException] {
    --- End diff --
    
    We need to update this test after we put a default value for `quantileProbabilities` and update its validator.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r39995988
  
    --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
    @@ -90,6 +108,9 @@ private[regression] trait AFTSurvivalRegressionParams extends Params
           SchemaUtils.checkColumnType(schema, $(censorCol), DoubleType)
           SchemaUtils.checkColumnType(schema, $(labelCol), DoubleType)
         }
    +    if (hasQuantileProbabilities) {
    --- End diff --
    
    `hasQuantilesCol && hasQuantileProbabilities`? If only one is set, I think we should make a warn log message.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141695981
  
    Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on a diff in the pull request:

    https://github.com/apache/spark/pull/8836#discussion_r40125924
  
    --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/AFTSurvivalRegressionSuite.scala ---
    @@ -235,10 +245,18 @@ class AFTSurvivalRegressionSuite extends SparkFunSuite with MLlibTestSparkContex
         model.setQuantileProbabilities(quantileProbabilities)
         assert(model.predictQuantiles(features) ~== quantilePredictR relTol 1E-3)
     
    -    model.transform(datasetMultivariate).select("features", "prediction").collect().foreach {
    -      case Row(features: DenseVector, prediction1: Double) =>
    -        val prediction2 = math.exp(BLAS.dot(model.coefficients, features) + model.intercept)
    +    model.setQuantilesCol("quantiles")
    +    model.transform(datasetMultivariate).select("features", "prediction", "quantiles")
    +      .collect().foreach {
    +      case Row(features: DenseVector, prediction1: Double, predictionQuantiles1: Vector) =>
    +        val lambda = math.exp(BLAS.dot(model.coefficients, features) + model.intercept)
    --- End diff --
    
    Ditto.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-141692998
  
      [Test build #42711 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42711/consoleFull) for   PR 8836 at commit [`1d8a682`](https://github.com/apache/spark/commit/1d8a6828cc7078b6cba229148e9da923f8507848).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142748168
  
    LGTM. Merged into master. Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142540772
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142183842
  
    Test PASSed.
    Refer to this link for build results (access rights to CI server needed): 
    https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/42813/
    Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142525194
  
    Merged build started.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] spark pull request: [SPARK-10686] [ML] Add quantilesCol to AFTSurv...

Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:

    https://github.com/apache/spark/pull/8836#issuecomment-142179284
  
     Merged build triggered.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org