kudu git commit: [spark] Avoid using TraversableLike.map in RowIterator.next()

[da...@apache.org]

Repository: kudu
Updated Branches:
  refs/heads/master 07e4cbd4d -> 2b413ee6c


[spark] Avoid using TraversableLike.map in RowIterator.next()

RowIterator.next() currently uses TraversableLike.map operation to
iterate over the columns in a Kudu RowResult. However, it's a generic
function that is not optimized for primitive integers, and profiling
revealed that it adds non-trivial overhead to executor performance.

By replacing it with Range.foreach (which is specialized for integers),
we could reduce the response times of single-column aggregation queries
by around 30%. e.g. "select max(col) from warmed_up_table"

Change-Id: I85ab882d5695ccb2fee09fe7a124129cf4610d56
Reviewed-on: http://gerrit.cloudera.org:8080/8922
Reviewed-by: Dan Burkert <da...@apache.org>
Tested-by: Kudu Jenkins


Project: http://git-wip-us.apache.org/repos/asf/kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/kudu/commit/2b413ee6
Tree: http://git-wip-us.apache.org/repos/asf/kudu/tree/2b413ee6
Diff: http://git-wip-us.apache.org/repos/asf/kudu/diff/2b413ee6

Branch: refs/heads/master
Commit: 2b413ee6cf20570f2ffdcf507bb6b0e8304176d9
Parents: 07e4cbd
Author: Junegunn Choi <ju...@gmail.com>
Authored: Fri Dec 29 10:37:24 2017 +0900
Committer: Dan Burkert <da...@apache.org>
Committed: Wed Jan 3 00:12:49 2018 +0000

----------------------------------------------------------------------
 .../src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala   | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/kudu/blob/2b413ee6/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
----------------------------------------------------------------------
diff --git a/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala b/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
index 3baa65f..daed3f0 100644
--- a/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
+++ b/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
@@ -128,6 +128,11 @@ private class RowIterator(private val scanner: KuduScanner,
 
   override def next(): Row = {
     val rowResult = currentIterator.next()
-    Row.fromSeq(Range(0, rowResult.getColumnProjection.getColumnCount).map(get(rowResult, _)))
+    val columnCount = rowResult.getColumnProjection.getColumnCount
+    val columns = Array.ofDim[Any](columnCount)
+    for (i <- 0 until columnCount) {
+      columns(i) = get(rowResult, i)
+    }
+    Row.fromSeq(columns)
   }
 }