You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by NarineK <gi...@git.apache.org> on 2016/02/12 06:13:27 UTC
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
GitHub user NarineK opened a pull request:
https://github.com/apache/spark/pull/11179
[SPARK-13295] [ ML, MLlib ] AFTSurvivalRegression.AFTAggregator improvements - Avoids creating new instances of arrays/vectors for each record
As also mentioned/marked by TODO in AFTAggregator.AFTAggregator.add(data: AFTPoint) a new array is being created for intercept value and it is being concatenated
with another array which contains the betas, the resulted Array is being converted into a Dense vector which in it's turn is being converted into breeze vector.
This is expensive and not necessarily beautiful.
I've tried to solve above mentioned problem by simple algebraic decompositions - keeping and treating intercept independently.
Please let me know what do you think and if you have any questions.
Thanks,
Narine
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/NarineK/spark survivaloptim
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11179.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11179
----
commit 8d443e9d7cd4b8b4cf7a4e14bec8287b7db6aff7
Author: Narine Kokhlikyan <na...@gmail.com>
Date: 2016-02-12T02:42:08Z
Initial commit - AFTSurvivalRegression improvements
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-185628757
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-184508061
**[Test build #51339 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51339/consoleFull)** for PR 11179 at commit [`e4707e7`](https://github.com/apache/spark/commit/e4707e775f34c0018f74451d048fb28a9c08ef48).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11179#discussion_r52870230
--- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
@@ -453,7 +455,8 @@ private class AFTAggregator(parameters: BDV[Double], fitIntercept: Boolean)
// Here we optimize loss function over beta and log(sigma)
def gradient: BDV[Double] = BDV.vertcat(BDV(Array(gradientLogSigmaSum / totalCnt.toDouble)),
- gradientBetaSum/totalCnt.toDouble)
+ BDV.vertcat(BDV(Array(gradientInterceptSum/totalCnt.toDouble)),
+ gradientBetaSum/totalCnt.toDouble))
--- End diff --
Use ```BDV.vertcat(BDV(Array(gradientLogSigmaSum / totalCnt.toDouble)), BDV(Array(gradientInterceptSum/totalCnt.toDouble)), gradientBetaSum/totalCnt.toDouble)``` to combine three parts directly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by NarineK <gi...@git.apache.org>.
Github user NarineK commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-184524951
Thank you for the review comments, @yanboliang
I've added your suggestions. Let me know if you have more comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-185090545
LGTM except minor issue, thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183205820
**[Test build #51173 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51173/consoleFull)** for PR 11179 at commit [`8d443e9`](https://github.com/apache/spark/commit/8d443e9d7cd4b8b4cf7a4e14bec8287b7db6aff7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183197817
**[Test build #51173 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51173/consoleFull)** for PR 11179 at commit [`8d443e9`](https://github.com/apache/spark/commit/8d443e9d7cd4b8b4cf7a4e14bec8287b7db6aff7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-184112663
@NarineK Thanks for working on this issue, it looks good overall and I left some inline comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11179#discussion_r53131871
--- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
@@ -437,23 +437,25 @@ object AFTSurvivalRegressionModel extends MLReadable[AFTSurvivalRegressionModel]
private class AFTAggregator(parameters: BDV[Double], fitIntercept: Boolean)
extends Serializable {
- // beta is the intercept and regression coefficients to the covariates
- private val beta = parameters.slice(1, parameters.length)
+ // the regression coefficients to the covariates
+ private val coefficients = parameters.slice(2, parameters.length)
+ private val intercept = parameters.valueAt(1)
// sigma is the scale parameter of the AFT model
private val sigma = math.exp(parameters(0))
private var totalCnt: Long = 0L
private var lossSum = 0.0
- private var gradientBetaSum = BDV.zeros[Double](beta.length)
+ private var gradientCoefficientSum = BDV.zeros[Double](coefficients.length)
+ private var gradientInterceptSum = 0.0
private var gradientLogSigmaSum = 0.0
def count: Long = totalCnt
def loss: Double = if (totalCnt == 0) 1.0 else lossSum / totalCnt
- // Here we optimize loss function over beta and log(sigma)
+ // Here we optimize loss function over coefficients and log(sigma)
--- End diff --
```beta``` means ```coefficients and intercept```, so here should be ```coefficients, intercept and log(sigma)``` in the annotation.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-185644709
LGTM, thanks @NarineK
ping @mengxr Could you take a look when you have time? [SPARK-13322](https://issues.apache.org/jira/browse/SPARK-13322) depends on this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-185605905
**[Test build #51474 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51474/consoleFull)** for PR 11179 at commit [`f2fbe34`](https://github.com/apache/spark/commit/f2fbe342b7302fbf5c80f94b775365baeff9c5fb).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-184515192
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11179#discussion_r52871118
--- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
@@ -481,8 +481,10 @@ private class AFTAggregator(parameters: BDV[Double], fitIntercept: Boolean)
assert(!lossSum.isInfinity,
s"AFTAggregator loss sum is infinity. Error for unknown reason.")
- gradientBetaSum += xi * (delta - math.exp(epsilon)) / sigma
- gradientLogSigmaSum += delta + (delta - math.exp(epsilon)) * epsilon
+ val delta_expeps = delta - math.exp(epsilon)
--- End diff --
```delta_expeps``` -> ```deltaMinusExpEps```, we use the camel case convention.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11179
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183206172
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183194055
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-185628462
**[Test build #51474 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51474/consoleFull)** for PR 11179 at commit [`f2fbe34`](https://github.com/apache/spark/commit/f2fbe342b7302fbf5c80f94b775365baeff9c5fb).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-184515101
**[Test build #51339 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51339/consoleFull)** for PR 11179 at commit [`e4707e7`](https://github.com/apache/spark/commit/e4707e775f34c0018f74451d048fb28a9c08ef48).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183193865
**[Test build #51170 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51170/consoleFull)** for PR 11179 at commit [`8d443e9`](https://github.com/apache/spark/commit/8d443e9d7cd4b8b4cf7a4e14bec8287b7db6aff7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-187464865
Merged into master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11179#discussion_r52869386
--- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
@@ -438,13 +438,15 @@ private class AFTAggregator(parameters: BDV[Double], fitIntercept: Boolean)
extends Serializable {
// beta is the intercept and regression coefficients to the covariates
--- End diff --
Please update the doc, beta is only the regression coefficients after your change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11179#discussion_r53295432
--- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
@@ -437,23 +437,25 @@ object AFTSurvivalRegressionModel extends MLReadable[AFTSurvivalRegressionModel]
private class AFTAggregator(parameters: BDV[Double], fitIntercept: Boolean)
extends Serializable {
- // beta is the intercept and regression coefficients to the covariates
- private val beta = parameters.slice(1, parameters.length)
+ // the regression coefficients to the covariates
+ private val coefficients = parameters.slice(2, parameters.length)
+ private val intercept = parameters.valueAt(1)
// sigma is the scale parameter of the AFT model
private val sigma = math.exp(parameters(0))
private var totalCnt: Long = 0L
private var lossSum = 0.0
- private var gradientBetaSum = BDV.zeros[Double](beta.length)
+ private var gradientCoefficientSum = BDV.zeros[Double](coefficients.length)
+ private var gradientInterceptSum = 0.0
private var gradientLogSigmaSum = 0.0
def count: Long = totalCnt
def loss: Double = if (totalCnt == 0) 1.0 else lossSum / totalCnt
- // Here we optimize loss function over beta and log(sigma)
+ // Here we optimize loss function over coefficients, intercept and log(sigma)
def gradient: BDV[Double] = BDV.vertcat(BDV(Array(gradientLogSigmaSum / totalCnt.toDouble)),
- gradientBetaSum/totalCnt.toDouble)
+ BDV(Array(gradientInterceptSum/totalCnt.toDouble)), gradientCoefficientSum/totalCnt.toDouble)
--- End diff --
Vector ```vertcat``` may be time-consuming, we can fix this issue use pre-assigned array. But it also involves other changes which are beyond the scope of this PR. I will fix this when solving [SPARK-13322](https://issues.apache.org/jira/browse/SPARK-13322), this PR is ready to merge, thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-185628760
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51474/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183194060
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51170/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by NarineK <gi...@git.apache.org>.
Github user NarineK commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-185599578
Fixed the comment, thnx @yanboliang
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183191566
@yanboliang Could you take a look?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11179#discussion_r52870569
--- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala ---
@@ -464,15 +467,12 @@ private class AFTAggregator(parameters: BDV[Double], fitIntercept: Boolean)
*/
def add(data: AFTPoint): this.type = {
- // TODO: Don't create a new xi vector each time.
- val xi = if (fitIntercept) {
- Vectors.dense(Array(1.0) ++ data.features.toArray).toBreeze
- } else {
- Vectors.dense(Array(0.0) ++ data.features.toArray).toBreeze
- }
+ val fitInterceptFlag = if (fitIntercept) 1.0 else 0.0
--- End diff --
```fitInterceptFlag``` -> ```interceptFlag```. ```fitInterceptFlag``` is more like a name of boolean variable.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183191594
ok to test
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183206177
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51173/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-183185609
**[Test build #51170 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51170/consoleFull)** for PR 11179 at commit [`8d443e9`](https://github.com/apache/spark/commit/8d443e9d7cd4b8b4cf7a4e14bec8287b7db6aff7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13295] [ ML, MLlib ] AFTSurvivalRegress...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11179#issuecomment-184515193
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51339/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org