You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2014/11/25 01:01:14 UTC
[jira] [Commented] (SPARK-4581) Refactorize StandardScaler to
improve the transformation performance
[ https://issues.apache.org/jira/browse/SPARK-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14223792#comment-14223792 ]
Apache Spark commented on SPARK-4581:
-------------------------------------
User 'dbtsai' has created a pull request for this issue:
https://github.com/apache/spark/pull/3435
> Refactorize StandardScaler to improve the transformation performance
> --------------------------------------------------------------------
>
> Key: SPARK-4581
> URL: https://issues.apache.org/jira/browse/SPARK-4581
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Reporter: DB Tsai
>
> The following optimizations are done to improve the StandardScaler model transformation performance.
> 1) Covert Breeze dense vector to primitive vector to reduce the overhead.
> 2) Since mean can be potentially a sparse vector, we explicitly convert it to dense primitive vector.
> 3) Have a local reference to `shift` and `factor` array so JVM can locate the value with one operation call.
> 4) In pattern matching part, we use the mllib SparseVector/DenseVector instead of breeze's vector to make the codebase cleaner.
> Benchmark with mnist8m dataset:
> Before,
> DenseVector withMean and withStd: 50.97secs
> DenseVector withMean and withoutStd: 42.11secs
> DenseVector withoutMean and withStd: 8.75secs
> SparseVector withoutMean and withStd: 5.437
> With this PR,
> DenseVector withMean and withStd: 5.76secs
> DenseVector withMean and withoutStd: 5.28secs
> DenseVector withoutMean and withStd: 5.30secs
> SparseVector withoutMean and withStd: 1.27
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org