You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/03 08:28:35 UTC
[GitHub] [spark] WeichenXu123 opened a new pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
WeichenXu123 opened a new pull request #30587:
URL: https://github.com/apache/spark/pull/30587
### What changes were proposed in this pull request?
Improve LogisticRegression test error tolerance
### Why are the changes needed?
When we switch BLAS version, some of the tests will fail due to too strict error tolerance in test.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
N/A
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737887682
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737873607
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36718/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #30587: [MINOR][ML] Increase Bounded MLOR (without regularization) test error tolerance
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738791366
I think it's OK as a band-aid and OK to back port
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on a change in pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on a change in pull request #30587:
URL: https://github.com/apache/spark/pull/30587#discussion_r535121974
##########
File path: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
##########
@@ -2962,7 +2962,14 @@ object LogisticRegressionSuite {
def checkCoefficientsEquivalent(coefficients1: Matrix, coefficients2: Matrix): Unit = {
coefficients1.colIter.zip(coefficients2.colIter).foreach { case (col1: Vector, col2: Vector) =>
(col1.asBreeze - col2.asBreeze).toArray.toSeq.sliding(2).foreach {
- case Seq(v1, v2) => assert(v1 ~= v2 absTol 1E-3)
+ case Seq(v1, v2) =>
+ if (math.abs(v1) < 1E-5 || math.abs(v2) < 1E-5) {
+ // If one of v1 and v2 is very close to zero, then compare absTol
+ assert(v1 ~= v2 absTol 1E-2, s"v1 ~= v2 absTol 0.01 faild: v1=${v1}, v2=${v2}")
+ }
+ else {
+ assert(v1 ~= v2 relTol 1E-2, s"v1 ~= v2 relTol 0.01 faild: v1=${v1}, v2=${v2}")
Review comment:
I guess you can keep previous logic `v1 ~= v2 absTol 1E-3` here.
##########
File path: mllib/src/test/scala/org/apache/spark/ml/classification/LogisticRegressionSuite.scala
##########
@@ -2962,7 +2962,14 @@ object LogisticRegressionSuite {
def checkCoefficientsEquivalent(coefficients1: Matrix, coefficients2: Matrix): Unit = {
coefficients1.colIter.zip(coefficients2.colIter).foreach { case (col1: Vector, col2: Vector) =>
(col1.asBreeze - col2.asBreeze).toArray.toSeq.sliding(2).foreach {
- case Seq(v1, v2) => assert(v1 ~= v2 absTol 1E-3)
+ case Seq(v1, v2) =>
+ if (math.abs(v1) < 1E-5 || math.abs(v2) < 1E-5) {
+ // If one of v1 and v2 is very close to zero, then compare absTol
+ assert(v1 ~= v2 absTol 1E-2, s"v1 ~= v2 absTol 0.01 faild: v1=${v1}, v2=${v2}")
+ }
Review comment:
nit: lint
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737877097
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738618522
**[Test build #132194 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132194/testReport)** for PR 30587 at commit [`7af0a12`](https://github.com/apache/spark/commit/7af0a12ab58bc5d56fa392d3710ddf86d42a7693).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737994983
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737748033
**[Test build #132107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132107/testReport)** for PR 30587 at commit [`e63fc10`](https://github.com/apache/spark/commit/e63fc106761e8732c7556611aa1176da927702e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737774332
**[Test build #132109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132109/testReport)** for PR 30587 at commit [`b3622fa`](https://github.com/apache/spark/commit/b3622fa1bd9161343cc7daa7d88e50cc2c46d377).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737908731
**[Test build #132127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132127/testReport)** for PR 30587 at commit [`ea1c58c`](https://github.com/apache/spark/commit/ea1c58c5137bf71c8cd37b0779df164fe03a325d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737858712
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36708/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738649588
@srowen Is this change ok now?
We need backport this to spark 3.1 branch, the adaptive blockify feature added in spark 3.1 increase the probability of this Bounded MLOR (without regularization) test failure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737815063
**[Test build #132109 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132109/testReport)** for PR 30587 at commit [`b3622fa`](https://github.com/apache/spark/commit/b3622fa1bd9161343cc7daa7d88e50cc2c46d377).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737871347
**[Test build #132117 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132117/testReport)** for PR 30587 at commit [`a51581d`](https://github.com/apache/spark/commit/a51581da8d8cb800b9a3365c73264a2d340816eb).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737768055
**[Test build #132107 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132107/testReport)** for PR 30587 at commit [`e63fc10`](https://github.com/apache/spark/commit/e63fc106761e8732c7556611aa1176da927702e1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738059888
@srowen
> a difference of 0.0001 vs 0.0002 could be significant.
One possibility is that, the result value will converge to 0, but in limited iterations, it may result in some near zero value like such as 1e-4 or 2e-4, @zhengruifeng Could you help check whether increase iteration will help make than converge fast enough to be zero ? Not sure the best threshold to set here for go into the branch of absTol test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738523827
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36770/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737833427
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737990079
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738622144
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36796/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 closed pull request #30587: [MINOR][ML] Increase Bounded MLOR (without regularization) test error tolerance
Posted by GitBox <gi...@apache.org>.
WeichenXu123 closed pull request #30587:
URL: https://github.com/apache/spark/pull/30587
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737752366
**[Test build #132108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132108/testReport)** for PR 30587 at commit [`07b71e2`](https://github.com/apache/spark/commit/07b71e2530c8de55f816140be33f636dc3276d94).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738523807
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36770/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737989640
**[Test build #132127 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132127/testReport)** for PR 30587 at commit [`ea1c58c`](https://github.com/apache/spark/commit/ea1c58c5137bf71c8cd37b0779df164fe03a325d).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737994983
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737772472
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738622144
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36796/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738484346
> If that's true, then isn't the 'right' expected value 0, and we're looking for a value at most some epsilon?
Reasonable. We should set expected value to be 0. Need check test code.
> If that's true, then isn't the 'right' expected value 0, and we're looking for a value at most some epsilon?
Resonable. Found some other issue in test code. Let me check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738495133
The master code use some werid logic to compare:
```
(col1.asBreeze - col2.asBreeze).toArray.toSeq.sliding(2).foreach {
case Seq(v1, v2) => assert(v1 ~= v2 absTol 1E-3)
}
```
change it to compare left and right directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737833427
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738591726
**[Test build #132194 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132194/testReport)** for PR 30587 at commit [`7af0a12`](https://github.com/apache/spark/commit/7af0a12ab58bc5d56fa392d3710ddf86d42a7693).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738522654
**[Test build #132169 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132169/testReport)** for PR 30587 at commit [`6632ac4`](https://github.com/apache/spark/commit/6632ac41cc96444af362c6a9c08e44078332f81a).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738504951
```
/**
* When no regularization is applied, the multinomial coefficients lack identifiability
* because we do not use a pivot class. We can add any constant value to the coefficients
* and get the same likelihood. If fitting under bound constrained optimization, we don't
* choose the mean centered coefficients like what we do for unbound problems, since they
* may out of the bounds. We use this function to check whether two coefficients are equivalent.
*/
def checkCoefficientsEquivalent(coefficients1: Matrix, coefficients2: Matrix): Unit = {
coefficients1.colIter.zip(coefficients2.colIter).foreach { case (col1: Vector, col2: Vector) =>
(col1.asBreeze - col2.asBreeze).toArray.toSeq.sliding(2).foreach {
case Seq(v1, v2) => assert(v1 ~= v2 absTol 1E-3)
}
}
}
```
According to the comments, it looks that this compare logic is designed on purpose. @WeichenXu123
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737752366
**[Test build #132108 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132108/testReport)** for PR 30587 at commit [`07b71e2`](https://github.com/apache/spark/commit/07b71e2530c8de55f816140be33f636dc3276d94).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738489746
@srowen @WeichenXu123
I had found that solver `BreezeLBFGSB` for Huber seems quite unstable, see [SPARK-32060](https://issues.apache.org/jira/browse/SPARK-32060) for details.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738622114
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36796/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738523069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132169/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737773121
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737908731
**[Test build #132127 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132127/testReport)** for PR 30587 at commit [`ea1c58c`](https://github.com/apache/spark/commit/ea1c58c5137bf71c8cd37b0779df164fe03a325d).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738611365
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36796/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738618905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132194/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738512549
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36770/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 edited a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
WeichenXu123 edited a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738059888
@srowen
> a difference of 0.0001 vs 0.0002 could be significant.
One possibility is that, the result value will converge to 0, but in limited iterations, it may result in some near zero value like such as 1e-4 or 2e-4, @zhengruifeng Could you help check whether increase iteration will help make than converge fast enough to be zero ? Not sure the best threshold to set here for go into the branch of absTol test.
Note: the probably failed test is the `multinomial logistic regression with intercept without regularization with bound` test, this test require to run Bounded-LBFGS algo (not LBFGS), this algo in breeze seems is buggy (sometimes unstable or converge slow), the algo is complex and hard to fix tho.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737990079
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738495042
**[Test build #132169 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132169/testReport)** for PR 30587 at commit [`6632ac4`](https://github.com/apache/spark/commit/6632ac41cc96444af362c6a9c08e44078332f81a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737773121
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737773151
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36707/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737887659
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36718/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng edited a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
zhengruifeng edited a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738489746
@srowen @WeichenXu123
I had found that solver `BreezeLBFGSB` for Huber seems quite unstable, see [SPARK-32060](https://issues.apache.org/jira/browse/SPARK-32060) for details.
When data is shuffled or blockified with different blocksize, `BreezeLBFGSB` output different solutions in my test for Huber.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737839663
**[Test build #132117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132117/testReport)** for PR 30587 at commit [`a51581d`](https://github.com/apache/spark/commit/a51581da8d8cb800b9a3365c73264a2d340816eb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737797791
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36707/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738495042
**[Test build #132169 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132169/testReport)** for PR 30587 at commit [`6632ac4`](https://github.com/apache/spark/commit/6632ac41cc96444af362c6a9c08e44078332f81a).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737994954
Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36728/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737772457
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 edited a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
WeichenXu123 edited a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738495133
The master code use some weird logic to compare:
```
(col1.asBreeze - col2.asBreeze).toArray.toSeq.sliding(2).foreach {
case Seq(v1, v2) => assert(v1 ~= v2 absTol 1E-3)
}
```
change it to compare left and right directly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738618905
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132194/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #30587: [MINOR][ML] Increase Bounded MLOR (without regularization) test error tolerance
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-740663359
@WeichenXu123 go ahead and merge and backport if you need it in 3.1
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738591726
**[Test build #132194 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132194/testReport)** for PR 30587 at commit [`7af0a12`](https://github.com/apache/spark/commit/7af0a12ab58bc5d56fa392d3710ddf86d42a7693).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 commented on pull request #30587: [MINOR][ML] Increase Bounded MLOR (without regularization) test error tolerance
Posted by GitBox <gi...@apache.org>.
WeichenXu123 commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-741500311
merge to master and branch-3.1, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737958517
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36728/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737748033
**[Test build #132107 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132107/testReport)** for PR 30587 at commit [`e63fc10`](https://github.com/apache/spark/commit/e63fc106761e8732c7556611aa1176da927702e1).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737877097
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737887682
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] WeichenXu123 edited a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
WeichenXu123 edited a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738059888
@srowen
> a difference of 0.0001 vs 0.0002 could be significant.
One possibility is that, the result value will converge to 0, but in limited iterations, it may result in some near zero value like such as 1e-4 or 2e-4, @zhengruifeng Could you help check whether increase iteration will help make than converge fast enough to be zero ? Not sure the best threshold to set here for go into the branch of absTol test.
Note: the probably failed test is the `multinomial logistic regression with intercept without regularization with bound` test, this test require to run Bounded-LBFGS algo (not LBFGS), this algo impl in breeze seems is buggy (sometimes unstable or converge slow), the algo is complex and hard to fix tho.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738523827
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/36770/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737839663
**[Test build #132117 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132117/testReport)** for PR 30587 at commit [`a51581d`](https://github.com/apache/spark/commit/a51581da8d8cb800b9a3365c73264a2d340816eb).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737772989
**[Test build #132108 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132108/testReport)** for PR 30587 at commit [`07b71e2`](https://github.com/apache/spark/commit/07b71e2530c8de55f816140be33f636dc3276d94).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737824703
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36708/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-737774332
**[Test build #132109 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/132109/testReport)** for PR 30587 at commit [`b3622fa`](https://github.com/apache/spark/commit/b3622fa1bd9161343cc7daa7d88e50cc2c46d377).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738523069
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/132169/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] srowen commented on pull request #30587: [MINOR][ML] Improve LogisticRegression test error tolerance
Posted by GitBox <gi...@apache.org>.
srowen commented on pull request #30587:
URL: https://github.com/apache/spark/pull/30587#issuecomment-738071093
If that's true, then isn't the 'right' expected value 0, and we're looking for a value at most some epsilon?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org