You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Hao Hao (Code Review)" <ge...@cloudera.org> on 2017/11/15 23:49:05 UTC
[kudu-CR] KUDU-1454 [part 2]: enable scan locality for Spark
Hao Hao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8560
Change subject: KUDU-1454 [part 2]: enable scan locality for Spark
......................................................................
KUDU-1454 [part 2]: enable scan locality for Spark
This patch provides support to take advantage of scan locality in Spark
integration, so that the scan will take place at the closet replica. It
also uses READ_AT_SNAPSHOT read mode without setting a timestamp to
ensure the consistency for these scans.
Change-Id: I5f0ea53c2c15ca05da141ff1763018854b84fcea
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
4 files changed, 41 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/60/8560/1
--
To view, visit http://gerrit.cloudera.org:8080/8560
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5f0ea53c2c15ca05da141ff1763018854b84fcea
Gerrit-Change-Number: 8560
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
[kudu-CR] KUDU-1454 [part 2]: enable scan locality for Spark
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8560 )
Change subject: KUDU-1454 [part 2]: enable scan locality for Spark
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8560/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala:
http://gerrit.cloudera.org:8080/#/c/8560/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala@52
PS1, Line 52: val SCAN_LOCALITY = "kudu.scanLocality"
Should this take CLOSEST/LEADER_ONLY instead of a bool so that we might add more in the future?
--
To view, visit http://gerrit.cloudera.org:8080/8560
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5f0ea53c2c15ca05da141ff1763018854b84fcea
Gerrit-Change-Number: 8560
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 16 Nov 2017 19:52:22 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1454 [part 2]: enable scan locality for Spark
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hello Dan Burkert, Kudu Jenkins, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/8560
to look at the new patch set (#2).
Change subject: KUDU-1454 [part 2]: enable scan locality for Spark
......................................................................
KUDU-1454 [part 2]: enable scan locality for Spark
This patch provides support to take advantage of scan locality in Spark
integration, so that the scan will take place at the closet replica. It
also uses READ_AT_SNAPSHOT read mode without setting a timestamp to
ensure the consistency for these scans.
Change-Id: I5f0ea53c2c15ca05da141ff1763018854b84fcea
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
4 files changed, 47 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/60/8560/2
--
To view, visit http://gerrit.cloudera.org:8080/8560
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I5f0ea53c2c15ca05da141ff1763018854b84fcea
Gerrit-Change-Number: 8560
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] KUDU-1454 [part 2]: enable scan locality for Spark
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/8560 )
Change subject: KUDU-1454 [part 2]: enable scan locality for Spark
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/8560
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5f0ea53c2c15ca05da141ff1763018854b84fcea
Gerrit-Change-Number: 8560
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Tue, 21 Nov 2017 21:12:29 +0000
Gerrit-HasComments: No
[kudu-CR] KUDU-1454 [part 2]: enable scan locality for Spark
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/8560 )
Change subject: KUDU-1454 [part 2]: enable scan locality for Spark
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/8560/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
File java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala:
http://gerrit.cloudera.org:8080/#/c/8560/1/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala@52
PS1, Line 52: val SCAN_LOCALITY = "kudu.scanLocality"
> Should this take CLOSEST/LEADER_ONLY instead of a bool so that we might add
Done
--
To view, visit http://gerrit.cloudera.org:8080/8560
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5f0ea53c2c15ca05da141ff1763018854b84fcea
Gerrit-Change-Number: 8560
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 17 Nov 2017 00:25:44 +0000
Gerrit-HasComments: Yes
[kudu-CR] KUDU-1454 [part 2]: enable scan locality for Spark
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8560 )
Change subject: KUDU-1454 [part 2]: enable scan locality for Spark
......................................................................
KUDU-1454 [part 2]: enable scan locality for Spark
This patch provides support to take advantage of scan locality in Spark
integration, so that the scan will take place at the closet replica. It
also uses READ_AT_SNAPSHOT read mode without setting a timestamp to
ensure the consistency for these scans.
Change-Id: I5f0ea53c2c15ca05da141ff1763018854b84fcea
Reviewed-on: http://gerrit.cloudera.org:8080/8560
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert <da...@apache.org>
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduContext.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
M java/kudu-spark/src/test/scala/org/apache/kudu/spark/kudu/DefaultSourceTest.scala
4 files changed, 47 insertions(+), 6 deletions(-)
Approvals:
Kudu Jenkins: Verified
Dan Burkert: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/8560
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5f0ea53c2c15ca05da141ff1763018854b84fcea
Gerrit-Change-Number: 8560
Gerrit-PatchSet: 3
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>