You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Hao Hao (Code Review)" <ge...@cloudera.org> on 2018/01/30 22:05:44 UTC
[kudu-CR] [spark] enable scan locality by default
Hao Hao has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9163
Change subject: [spark] enable scan locality by default
......................................................................
[spark] enable scan locality by default
Commit 3abca98c5 introduced support to take advantage of scan locality
in Spark integration, however this feature is not enabled by default.
This patch defaults scan locality to use closest replica. And if the
client configures to not to use the default, then only list leader
replica as the perferred location to schedule a task.
Change-Id: Ifc94e9ce52877a88f96a06599a8657da929b6126
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
2 files changed, 12 insertions(+), 4 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/63/9163/1
--
To view, visit http://gerrit.cloudera.org:8080/9163
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ifc94e9ce52877a88f96a06599a8657da929b6126
Gerrit-Change-Number: 9163
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
[kudu-CR] [spark] enable scan locality by default
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/9163 )
Change subject: [spark] enable scan locality by default
......................................................................
Patch Set 1:
> I'm a little nervous that this is going to result in normal
> applications losing RYW consistency, is that a valid concern?
Hmm, why do you think so? That is why in previous patch READ_AT_SNAPSHOT mode is used when closest replica is selected: https://github.com/apache/kudu/blob/master/java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala#L56
--
To view, visit http://gerrit.cloudera.org:8080/9163
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc94e9ce52877a88f96a06599a8657da929b6126
Gerrit-Change-Number: 9163
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 01 Feb 2018 19:32:30 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] enable scan locality by default
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/9163 )
Change subject: [spark] enable scan locality by default
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/9163
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc94e9ce52877a88f96a06599a8657da929b6126
Gerrit-Change-Number: 9163
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Fri, 16 Feb 2018 18:59:43 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] enable scan locality by default
Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9163 )
Change subject: [spark] enable scan locality by default
......................................................................
[spark] enable scan locality by default
Commit 3abca98c5 introduced support to take advantage of scan locality
in Spark integration, however this feature is not enabled by default.
This patch defaults scan locality to use closest replica. And if the
client configures to not to use the default, then only list leader
replica as the perferred location to schedule a task.
Change-Id: Ifc94e9ce52877a88f96a06599a8657da929b6126
Reviewed-on: http://gerrit.cloudera.org:8080/9163
Tested-by: Kudu Jenkins
Reviewed-by: Dan Burkert <da...@cloudera.com>
---
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/DefaultSource.scala
M java/kudu-spark/src/main/scala/org/apache/kudu/spark/kudu/KuduRDD.scala
2 files changed, 12 insertions(+), 4 deletions(-)
Approvals:
Kudu Jenkins: Verified
Dan Burkert: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/9163
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifc94e9ce52877a88f96a06599a8657da929b6126
Gerrit-Change-Number: 9163
Gerrit-PatchSet: 2
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] [spark] enable scan locality by default
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/9163 )
Change subject: [spark] enable scan locality by default
......................................................................
Patch Set 1:
I'm a little nervous that this is going to result in normal applications losing RYW consistency, is that a valid concern?
--
To view, visit http://gerrit.cloudera.org:8080/9163
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc94e9ce52877a88f96a06599a8657da929b6126
Gerrit-Change-Number: 9163
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 01 Feb 2018 19:16:06 +0000
Gerrit-HasComments: No
[kudu-CR] [spark] enable scan locality by default
Posted by "Dan Burkert (Code Review)" <ge...@cloudera.org>.
Dan Burkert has posted comments on this change. ( http://gerrit.cloudera.org:8080/9163 )
Change subject: [spark] enable scan locality by default
......................................................................
Patch Set 1:
Ah, ok great, I think I missed that. So the READ_AT_SNAPSHOT with the propagated TS should be enough to ensure RYW. Makes sense.
--
To view, visit http://gerrit.cloudera.org:8080/9163
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifc94e9ce52877a88f96a06599a8657da929b6126
Gerrit-Change-Number: 9163
Gerrit-PatchSet: 1
Gerrit-Owner: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 01 Feb 2018 19:43:20 +0000
Gerrit-HasComments: No