You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/02/06 06:53:44 UTC
[GitHub] [spark] zhengruifeng opened a new pull request #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
zhengruifeng opened a new pull request #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473
### What changes were proposed in this pull request?
1, add Level-2 BLAS routine `ger`;
2, stack input vectors to blocks to use Level-3 BLAS routine in training
### Why are the changes needed?
for performance:
1, save about 40% RAM;
2, 25% ~ 60% faster; 30% ~ 60% faster with openBLAS
### Does this PR introduce any user-facing change?
add a new expert param `blockSize`
### How was this patch tested?
added testsuites
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng closed pull request #27473:
[SPARK-30699][WIP][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
zhengruifeng closed pull request #27473: [SPARK-30699][WIP][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng commented on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
zhengruifeng commented on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582765435
data: `a9a`: numFeatures=123, numInstances=32,561
testCode:
```scala
import org.apache.spark.ml.clustering._
import org.apache.spark.storage.StorageLevel
import org.apache.spark.ml.linalg._
val df = spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a")
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count
new GaussianMixture().fit(df)
val results = Seq(2, 8, 32).map { k => val start = System.currentTimeMillis; val model = new GaussianMixture().setK(k).setSeed(0).setTol(0).setMaxIter(20).fit(df); val dur = System.currentTimeMillis - start; (model, dur) }
results.map(_._2)
results.map(_._1.summary.numIter)
results.map(_._1.summary.logLikelihood)
results.map(t => t._2.toDouble / t._1.summary.numIter)
```
result:
|Impl| This PR(k=2) | This PR(k=8) | This PR(k=32) | Master(k=2) | Master(k=8) | Master(k=32) |
|------|----------|------------|----------|------------|----------|----------|
|numIter|3|3|5|3|8|4|
|logLikelihood|2814835.470027733|2817523.6371536762|2820228.372200876|2814835.4700277536|2817523.6371535994|2820228.3722007386|
|dur per iteration| 950.0 | 2744.3333333333335 | 9780.6 | 1192.6666666666667 | 3446.375| 15724.75 |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582769600
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582765767
**[Test build #117976 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117976/testReport)** for PR 27473 at commit [`448ed80`](https://github.com/apache/spark/commit/448ed80bf72c56cfa7b25f5174f9e674ab22ddcf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582764139
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22740/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582764139
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/22740/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582769604
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117976/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582769600
Merged build finished. Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582764132
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582764132
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng edited a comment on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
zhengruifeng edited a comment on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582765435
data: `a9a`: numFeatures=123, numInstances=32,561
testCode:
```scala
import org.apache.spark.ml.clustering._
import org.apache.spark.storage.StorageLevel
import org.apache.spark.ml.linalg._
val df = spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a")
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count
new GaussianMixture().fit(df)
val results = Seq(2, 8, 32).map { k => val start = System.currentTimeMillis; val model = new GaussianMixture().setK(k).setSeed(0).setTol(0).setMaxIter(20).fit(df); val dur = System.currentTimeMillis - start; (model, dur) }
results.map(_._2)
results.map(_._1.summary.numIter)
results.map(_._1.summary.logLikelihood)
results.map(t => t._2.toDouble / t._1.summary.numIter)
```
result:
|Impl| This PR(k=2) | This PR(k=8) | This PR(k=32) | Master(k=2) | Master(k=8) | Master(k=32) |
|------|----------|------------|----------|------------|----------|----------|
|numIter|3|3|5|3|8|4|
|logLikelihood|2814835.470027733|2817523.6371536762|2820228.372200876|2814835.4700277536|2817523.6371535994|2820228.3722007386|
|dur per iteration| 950.0 | 2744.3333333333335 | 9780.6 | 1192.6666666666667 | 3446.375| 15724.75 |
With NativeBLAS with OPENBLAS_NUM_THREADS=1:
|Impl| This PR(k=2) | This PR(k=8) | This PR(k=32) | Master(k=2) | Master(k=8) | Master(k=32) |
|------|----------|------------|----------|------------|----------|----------|
|numIter|3|3|20|3|20|8|
|logLikelihood|2814835.4700277243 | 2817523.6371536693 | 2820228.372200879|2814835.4700277452|2817523.6371535957|2820228.372200736|
|dur per iteration| 780.0| 2356.3333333333335 | 8192.95 | 1183.0 | 3022.1 | 13073.75 |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582769567
**[Test build #117976 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117976/testReport)** for PR 27473 at commit [`448ed80`](https://github.com/apache/spark/commit/448ed80bf72c56cfa7b25f5174f9e674ab22ddcf).
* This patch **fails MiMa tests**.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582769604
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/117976/
Test FAILed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582765767
**[Test build #117976 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/117976/testReport)** for PR 27473 at commit [`448ed80`](https://github.com/apache/spark/commit/448ed80bf72c56cfa7b25f5174f9e674ab22ddcf).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] zhengruifeng edited a comment on issue #27473:
[SPARK-30699][ML][PYSPARK] GMM blockify input vectors
Posted by GitBox <gi...@apache.org>.
zhengruifeng edited a comment on issue #27473: [SPARK-30699][ML][PYSPARK] GMM blockify input vectors
URL: https://github.com/apache/spark/pull/27473#issuecomment-582765435
data: `a9a`: numFeatures=123, numInstances=32,561
testCode:
```scala
import org.apache.spark.ml.clustering._
import org.apache.spark.storage.StorageLevel
import org.apache.spark.ml.linalg._
val df = spark.read.format("libsvm").load("/data1/Datasets/a9a/a9a")
df.persist(StorageLevel.MEMORY_AND_DISK)
df.count
new GaussianMixture().fit(df)
val results = Seq(2, 8, 32).map { k => val start = System.currentTimeMillis; val model = new GaussianMixture().setK(k).setSeed(0).setTol(0).setMaxIter(20).fit(df); val dur = System.currentTimeMillis - start; (model, dur) }
results.map(_._2)
results.map(_._1.summary.numIter)
results.map(_._1.summary.logLikelihood)
results.map(t => t._2.toDouble / t._1.summary.numIter)
```
result:
|Impl| This PR(k=2) | This PR(k=8) | This PR(k=32) | Master(k=2) | Master(k=8) | Master(k=32) |
|------|----------|------------|----------|------------|----------|----------|
|numIter|3|3|5|3|8|4|
|logLikelihood|2814835.470027733|2817523.6371536762|2820228.372200876|2814835.4700277536|2817523.6371535994|2820228.3722007386|
|dur per iteration| 950.0 | 2744.3333333333335 | 9780.6 | 1192.6666666666667 | 3446.375| 15724.75 |
With NativeBlas NativeBlas OPENBLAS_NUM_THREADS=1:
|Impl| This PR(k=2) | This PR(k=8) | This PR(k=32) | Master(k=2) | Master(k=8) | Master(k=32) |
|------|----------|------------|----------|------------|----------|----------|
|numIter|3|3|20|3|20|8|
|logLikelihood|2814835.4700277243 | 2817523.6371536693 | 2820228.372200879|2814835.4700277452|2817523.6371535957|2820228.372200736|
|dur per iteration| 780.0| 2356.3333333333335 | 8192.95 | 1183.0 | 3022.1 | 13073.75 |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org