[jira] [Commented] (SPARK-19759) ALSModel.predict on Dataframes : potential optimization by not using blas

["Marco Gaido (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-19759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16242314#comment-16242314 ] 

Marco Gaido commented on SPARK-19759:
-------------------------------------

I tried comparing the current implementation with an easy for loop to compute directly the dot product on 500.000 elements with 100 features. The code I used is:

{code}
def time[R](block: => R): Unit = {
    val t0 = System.nanoTime()
    block
    val t1 = System.nanoTime()
    println("Elapsed time: " + (t1 - t0) + "ns")
}
val r = new scala.util.Random(100)
val input = (1 to 500000).map(_ => (1 to 100).map(_ => r.nextFloat).toSeq)
def f(a:Seq[Float], b:Seq[Float]): Float = {
    var r = 0.0f
    for(i <- 0 until a.length) {
        r+=a(i)*b(i)
    }
    r
}
import com.github.fommil.netlib.BLAS.{getInstance => blas}
val b = (1 to 100).map(_ => r.nextFloat).toSeq
time { input.foreach(a=>blas.sdot(100, a.toArray, 1, b.toArray, 1)) }
time { input.foreach(a=>f(a,b)) }
{code}

On 10 run the current implementation takes on average 2968.718815 ms, while the for-loop implementation takes 515.510185 ms.
Thus I am submitting a PR with this second implementation.

> ALSModel.predict on Dataframes : potential optimization by not using blas 
> --------------------------------------------------------------------------
>
>                 Key: SPARK-19759
>                 URL: https://issues.apache.org/jira/browse/SPARK-19759
>             Project: Spark
>          Issue Type: Improvement
>          Components: ML
>    Affects Versions: 2.1.1
>            Reporter: Sue Ann Hong
>            Priority: Minor
>
> In the DataFrame ALS prediction function, we use blas.sdot which may be slower due to the conversion to Arrays. We can try operating on Seqs or another data structure to see if avoiding the conversion makes the operation faster. Ref: https://github.com/apache/spark/pull/17090/files/707bc6b153a7f899fbf3fe2a5675cacba1f95711#diff-be65dd1d6adc53138156641b610fcada 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org