[jira] [Created] (SPARK-4581) Refactorize StandardScaler to improve the transformation performance

["DB Tsai (JIRA)" <ji...@apache.org>]

DB Tsai created SPARK-4581:
------------------------------

             Summary: Refactorize StandardScaler to improve the transformation performance
                 Key: SPARK-4581
                 URL: https://issues.apache.org/jira/browse/SPARK-4581
             Project: Spark
          Issue Type: Improvement
          Components: MLlib
            Reporter: DB Tsai


The following optimizations are done to improve the StandardScaler model transformation performance.

1) Covert Breeze dense vector to primitive vector to reduce the overhead.
2) Since mean can be potentially a sparse vector, we explicitly convert it to dense primitive vector.
3) Have a local reference to `shift` and `factor` array so JVM can locate the value with one operation call.
4) In pattern matching part, we use the mllib SparseVector/DenseVector instead of breeze's vector to make the codebase cleaner. 

Benchmark with mnist8m dataset:

Before,
DenseVector withMean and withStd: 50.97secs
DenseVector withMean and withoutStd: 42.11secs
DenseVector withoutMean and withStd: 8.75secs
SparseVector withoutMean and withStd: 5.437

With this PR,
DenseVector withMean and withStd: 5.76secs
DenseVector withMean and withoutStd: 5.28secs
DenseVector withoutMean and withStd: 5.30secs
SparseVector withoutMean and withStd: 1.27



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org