You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Junegunn Choi (Code Review)" <ge...@cloudera.org> on 2017/12/29 01:53:55 UTC
[kudu-CR] [spark] Avoid using TraversableLike.map in RowIterator.next()
Junegunn Choi has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8922
Change subject: [spark] Avoid using TraversableLike.map in RowIterator.next()
......................................................................
[spark] Avoid using TraversableLike.map in RowIterator.next()
RowIterator.next() currently uses TraversableLike.map operation to
iterate over the columns in a Kudu RowResult. However, it's a generic
function that is not optimized for primitive integers, and profiling
revealed that it adds non-trivial overhead to executor performance.
By replacing it with Range.foreach (which is specialized for integers),
we could reduce the response times of single-column aggregation queries
by around 30%. e.g. "select max(col) from warmed_up_table"
Change-Id: I85ab882d5695ccb2fee09fe7a124129cf4610d56
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
1 file changed, 6 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/22/8922/1
--
To view, visit http://gerrit.cloudera.org:8080/8922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I85ab882d5695ccb2fee09fe7a124129cf4610d56
Gerrit-Change-Number: 8922
Gerrit-PatchSet: 1
Gerrit-Owner: Junegunn Choi <ju...@gmail.com>
[kudu-CR] [spark] Avoid using TraversableLike.map in RowIterator.next()
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8922 )
Change subject: [spark] Avoid using TraversableLike.map in RowIterator.next()
......................................................................
[spark] Avoid using TraversableLike.map in RowIterator.next()
RowIterator.next() currently uses TraversableLike.map operation to
iterate over the columns in a Kudu RowResult. However, it's a generic
function that is not optimized for primitive integers, and profiling
revealed that it adds non-trivial overhead to executor performance.
By replacing it with Range.foreach (which is specialized for integers),
we could reduce the response times of single-column aggregation queries
by around 30%. e.g. "select max(col) from warmed_up_table"
Change-Id: I85ab882d5695ccb2fee09fe7a124129cf4610d56
Reviewed-on: http://gerrit.cloudera.org:8080/8922
Reviewed-by: Dan Burkert <da...@apache.org>
Tested-by: Kudu Jenkins
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
1 file changed, 6 insertions(+), 1 deletion(-)
Approvals:
Dan Burkert: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/8922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I85ab882d5695ccb2fee09fe7a124129cf4610d56
Gerrit-Change-Number: 8922
Gerrit-PatchSet: 2
Gerrit-Owner: Junegunn Choi <ju...@gmail.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] [spark] Avoid using TraversableLike.map in RowIterator.next()
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8922 )
Change subject: [spark] Avoid using TraversableLike.map in RowIterator.next()
......................................................................
Patch Set 1: Code-Review+2
Nice find. The Scala collections library never ceases to disappoint.
--
To view, visit http://gerrit.cloudera.org:8080/8922
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I85ab882d5695ccb2fee09fe7a124129cf4610d56
Gerrit-Change-Number: 8922
Gerrit-PatchSet: 1
Gerrit-Owner: Junegunn Choi <ju...@gmail.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Comment-Date: Tue, 02 Jan 2018 22:28:13 +0000
Gerrit-HasComments: No