[GitHub] [spark] zhengruifeng commented on pull request #31588: [SPARK-34470][ML] VectorSlicer use ordering if possible

[GitBox <gi...@apache.org>]

zhengruifeng commented on pull request #31588:
URL: https://github.com/apache/spark/pull/31588#issuecomment-781877440


   test:
   ```
     test("performance") {
       val rng = new Random(123)
       val n = 100000
       val dim = 10000
       val nnz = 100
       val vectors = Array.tabulate(n) { i =>
         val indices = rng.shuffle(Seq.range(0, dim)).take(nnz).sorted.toArray
         val values = Array.fill(nnz)(rng.nextGaussian)
         new SparseVector(dim, indices, values)
       }
   
       val slicingIndices = rng.shuffle(Seq.range(0, dim)).take(nnz).sorted.toArray
   
       val tic0 = System.currentTimeMillis()
       Seq.range(0, 100).foreach { i =>
         vectors.foreach { sv => sv.slice(slicingIndices) }
       }
       val toc0 = System.currentTimeMillis()
       println(s"slice: ${toc0 - tic0}")
   
       val tic1 = System.currentTimeMillis()
       Seq.range(0, 100).foreach { i =>
         vectors.foreach { sv => sv.sliceSorted(slicingIndices) }
       }
       val toc1 = System.currentTimeMillis()
       println(s"sliceSorted: ${toc1 - tic1}")
     }
   ```
   
   results:
   
   ![image](https://user-images.githubusercontent.com/7322292/108469701-b6714d80-72c3-11eb-9d6b-86f2c6130383.png)
   
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org