You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by fe...@apache.org on 2017/11/12 22:45:01 UTC
spark git commit: [SPARK-21694][R][ML] Reduce max iterations in
Linear SVM test in R to speed up AppVeyor build
Repository: spark
Updated Branches:
refs/heads/branch-2.2 2a04cfaad -> 8acd02f42
[SPARK-21694][R][ML] Reduce max iterations in Linear SVM test in R to speed up AppVeyor build
This PR proposes to reduce max iteration in Linear SVM test in SparkR. This particular test elapses roughly 5 mins on my Mac and over 20 mins on Windows.
The root cause appears, it triggers 2500ish jobs by the default 100 max iterations. In Linux, `daemon.R` is forked but on Windows another process is launched, which is extremely slow.
So, given my observation, there are many processes (not forked) ran on Windows, which makes the differences of elapsed time.
After reducing the max iteration to 10, the total jobs in this single test is reduced to 550ish.
After reducing the max iteration to 5, the total jobs in this single test is reduced to 360ish.
Manually tested the elapsed times.
Author: hyukjinkwon <gu...@gmail.com>
Closes #19722 from HyukjinKwon/SPARK-21693-test.
(cherry picked from commit 3d90b2cb384affe8ceac9398615e9e21b8c8e0b0)
Signed-off-by: Felix Cheung <fe...@apache.org>
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8acd02f4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8acd02f4
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8acd02f4
Branch: refs/heads/branch-2.2
Commit: 8acd02f42fa111d8e5177fd9cfc74b5c1c18fb63
Parents: 2a04cfa
Author: hyukjinkwon <gu...@gmail.com>
Authored: Sun Nov 12 14:37:20 2017 -0800
Committer: Felix Cheung <fe...@apache.org>
Committed: Sun Nov 12 14:44:36 2017 -0800
----------------------------------------------------------------------
R/pkg/tests/fulltests/test_mllib_classification.R | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/8acd02f4/R/pkg/tests/fulltests/test_mllib_classification.R
----------------------------------------------------------------------
diff --git a/R/pkg/tests/fulltests/test_mllib_classification.R b/R/pkg/tests/fulltests/test_mllib_classification.R
index 726e9d9..892d8a6 100644
--- a/R/pkg/tests/fulltests/test_mllib_classification.R
+++ b/R/pkg/tests/fulltests/test_mllib_classification.R
@@ -66,10 +66,9 @@ test_that("spark.svmLinear", {
feature <- c(1.1419053, 0.9194079, -0.9498666, -1.1069903, 0.2809776)
data <- as.data.frame(cbind(label, feature))
df <- createDataFrame(data)
- model <- spark.svmLinear(df, label ~ feature, regParam = 0.1)
+ model <- spark.svmLinear(df, label ~ feature, regParam = 0.1, maxIter = 5)
prediction <- collect(select(predict(model, df), "prediction"))
expect_equal(sort(prediction$prediction), c("0.0", "0.0", "0.0", "1.0", "1.0"))
-
})
test_that("spark.logit", {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org