You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Alexey Serbin (Code Review)" <ge...@cloudera.org> on 2018/12/21 01:31:04 UTC
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Alexey Serbin has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12119
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2018-12-20-location-awareness.md
1 file changed, 155 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/1
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newchange
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Greg Solovyev (Code Review)" <ge...@cloudera.org>.
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 5:
What about a link to a sample script or an inline sample script in the post?
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Jan 2019 00:44:50 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Greg Solovyev, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#8).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2019-03-25-location-awareness.md
1 file changed, 348 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/8
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Greg Solovyev,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#5).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2018-12-20-location-awareness.md
1 file changed, 284 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/5
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 5:
> What about a link to a sample script or an inline sample script in
> the post?
Whoops, I need to add that, sure. I'll do that tonight. Thanks for the reminder!
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Jan 2019 02:41:46 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 8:
(28 comments)
Thank you for the thorough review.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md
File _posts/2019-03-25-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@1
PS8, Line 1: ---
> Can you push this to the gh_pages branch in your github fork so a rendered
Done: https://github.com/alexeyserbin/kudu/blob/gh-pages/_posts/2019-03-25-location-awareness.md
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@15
PS8, Line 15: <!--TODO(aserbin) rename the file to reflect the date when published -->
> Should this be removed?
I'll remove it once I'm getting +1 or +2 on the latest patchset.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@19
PS8, Line 19: first cut
> initial implementation?
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@19
PS8, Line 19: starting 1.9.0
> ...starting *with the* 1.9.0...
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@20
PS8, Line 20: is built for the following use case:
> I am not sure this is a "use case" per se, but instead what the term "locat
I rephrased it using term 'requirement'. Does it sound better now?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@26
PS8, Line 26: A rack failure might happen because of a failure of a hardware component shared
: among servers in the rack: network switch, power supply, etc.
> A rack failure can occur when a hardware component shared
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@31
PS8, Line 31: network latency between datacenters is low.
> This is a good opportunity to explicitly mention that this is why we call t
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@37
PS8, Line 37: are
: supposed to
> should
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@38
PS8, Line 38: physical or cloud-defined hierarchy of the
: deployed cluster
> I am not sure I understand what this means in relation to location awarenes
I recall trying to introduce 'fault domain' term in this blog , I got feedback that we better not introduce it since 1) readers might be not aware of that concept, 2) we don't use that term in how design doc. That's why you don't see 'failure domain' or 'fault domain' here at all. If you feel strong about the necessity to speak in 'proper' terms here, let me know and I'll introduce those terms with corresponding references.
As of now, I just added the examples you suggested.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@41
PS8, Line 41: However, we want to keep the hierarchy
: there to make it possible to exploit it later
> However, we plan to leverage the hierarchical structure in future releases.
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@43
PS8, Line 43: compatibility with HDFS
> Perhaps this should be moved up and describe a bit more in detail as a desi
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@52
PS8, Line 52: etc
> What is the "etc"? What else does it use it for?
The text below contains information about what those locations are used for, so 'etc' here is for:
* client finding the closest tablet server to its own location
* other usages of the location hierarchy which we haven't leveraged yet
I removed the 'etc'. Does it look better to you now?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@55
PS8, Line 55: location string for the specified IP address/hostname.
> The script below specifically shows ip-address. How do I use hostname?
The script below is an example. You are free to use hostname to perform any flavor of mapping 'hostname' --> 'location string' you want. I added corresponding note into the example's comment.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@59
PS8, Line 59: tablet server restarts
> Is this dependent on `--follower_unavailable_considered_failed_sec`? Or wil
It's not dependent of '--follower_unavailable_considered_failed_sec'. That's about registering tablet servers, not tracking tablet replicas. Any restart of a tablet server will cause assignment of location to the tablet server.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@59
PS8, Line 59: Kudu tablet servers are location
: agnostic, at least for now, so the assigned location is not reported back
: to the tablet server.
> Maybe this paragraph would flow better if you moved this part to the bottom
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@64
PS8, Line 64: masters provide connected clients
> How do they do this?
I added a reference to the design doc.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@62
PS8, Line 62: to try to place replicas evenly across
: locations and to keep tablets available in case all tablet servers in a single
: location fail.
> This last part is somewhat duplicated from the Introduction section above.
I want to keep it here because it's a bit different context, and I think this is some useful sort of 'duplication' that allows to grasp the presented information better.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@75
PS8, Line 75: Essentially, that's about having
> This results in...
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@81
PS8, Line 81: The error handling and the input validation are minimalistic. Also, the
: # network topology choice, supportability and capacity planning aspects of
: # this script might be sub-optimal if applied as-is for real-world use cases.
> Is there anywhere else anyone can get a "good" production worthy example? I
I don't think we should aim for a 'good' production worthy example in this blog because too many error handling and other details will make it harder to read and understand.
This is an example, and the deficiencies of this script are outlined. Readers interested in extending this example and making it closer to 'good' (what's 'good', BTW?) production-grade script can do that to their taste.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@104
PS8, Line 104: echo "ERROR: '$ip_address' is not a valid IPv4 address"
> Should errors map to "/other"? How does Kudu handle this script exiting wit
I think errors should map to errors/failures. That way it's simpler to track and fix them. If the location mapping script returns an error, the tablet server's registration fails and the tablet server is not added into the master's registry, making it inaccessible to clients. I added information on what happens if location assignment script fails into the text of the blog and into this example script.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@142
PS8, Line 142: The reasoning is simple: with
> I try to stay away from saying something is "simple". People have wide leve
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@151
PS8, Line 151: The location-aware placement policy for tablet replicas in Kudu
> This seems more appropriate for earlier sections. When reading the blog pos
Indeed. I re-shuffled the paragraphs and explicitly mentioned that the policy is the only available one and it doesn't have any parameters to configure.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@162
PS8, Line 162: Automatic re-replication and placement policy
> Per my earlier comment, this is also more about "How it works".
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@177
PS8, Line 177: Reinstating location-aware policy in Kudu cluster
> I think this is "How to use it" and makes sense here.
Ack
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@208
PS8, Line 208: Examples
> Per my earlier comment, this is also more about "How it works".
Done
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@337
PS8, Line 337: roadmap
> What roadmap? Does Apache Kudu have a roadmap?
I think the same roadmap that people refer when asking questions like 'are you going to implement feature A any soon?'.
There is a request to implement some sort of 'tagging' as JIRA item at issues.apache.org. I think I'll just add this here.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@342
PS8, Line 342: see [2]
> Any reason not to link inline instead of using reference style?
Nothing particular, but I was hoping to get it referenced as a link in [2] as well. Probably, I confused .md format with something else.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@346
PS8, Line 346: [[1]] [Location awareness in Kudu, design document](https://s.apache.org/location-awareness-design)
> Can we check this design doc into https://github.com/apache/kudu/tree/maste
Yes, but it will take some time to convert the design doc into .md format and pass it through review cycles. It should be done anyway, so I'll try to do so shortly.
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 26 Mar 2019 18:06:02 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Greg Solovyev (Code Review)" <ge...@cloudera.org>.
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 7: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 7
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Jan 2019 00:27:08 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 5:
(2 comments)
cool post, enjoyed catching up on what you've accomplished with this feature
http://gerrit.cloudera.org:8080/#/c/12119/5/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/5/_posts/2018-12-20-location-awareness.md@1
PS5, Line 1: ---
just a reminder: should rename the file to the current date before committing so it shows up as new when it's posted on the blog
http://gerrit.cloudera.org:8080/#/c/12119/5/_posts/2018-12-20-location-awareness.md@28
PS5, Line 28: replace 'rack' with any other aggregation of nodes (e.g., chassis, site, etc.)
might be good to add "cloud availability zone" to this list
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Mon, 28 Jan 2019 23:15:50 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 5:
(2 comments)
> (2 comments)
>
> cool post, enjoyed catching up on what you've accomplished with
> this feature
Yep, Will and I did some progress on that since the LA project started. Fengling contributed few patches as well. That's a team effort :)
Thank you for the review!
http://gerrit.cloudera.org:8080/#/c/12119/5/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/5/_posts/2018-12-20-location-awareness.md@1
PS5, Line 1: ---
> just a reminder: should rename the file to the current date before committi
Thank you for the reminder -- I added a TODO for that.
http://gerrit.cloudera.org:8080/#/c/12119/5/_posts/2018-12-20-location-awareness.md@28
PS5, Line 28: replace 'rack' with any other aggregation of nodes (e.g., chassis, site, etc.)
> might be good to add "cloud availability zone" to this list
Done
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 30 Jan 2019 00:12:04 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Greg Solovyev, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#7).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2018-12-20-location-awareness.md
1 file changed, 349 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/7
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 7
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 11:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/12119/11//COMMIT_MSG
Commit Message:
PS11:
Rendered version is at:
https://gist.github.com/alexeyserbin/a6049124e63f32a8cf344a3fe7c55489
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md
File _posts/2019-04-16-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@48
PS10, Line 48:
:
> Is there a Jira tracking this?
It will be useful in context of KUDU-2604.
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@62
PS10, Line 62:
:
> How should users detect this? Does the tablet server fail? Is there a log t
As one criterion, non-registered tablet servers does not appear in the list of tablet servers at masters' web UI. Also, masters will not return addresses of non-registered tablet servers in the ListTabletServers -- that makes them unusable to Kudu clients.
Of course masters log about inability to determine location for a table server.
I added a few details about that.
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@73
PS10, Line 73:
> client's assigned
Done
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@81
PS10, Line 81:
> follow the following just sounds funny because it's two forms for "follow".
That sounds funny, indeed :)
Done
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@85
PS10, Line 85:
> as the replica placement policy
Done
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@87
PS10, Line 87:
> remove "the"
Done
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@103
PS10, Line 103:
> Is thee a jira to track this?
I opened a new one upstream: KUDU-2780
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@368
PS10, Line 368:
> Now that these are linked inline, are they needed here too?
I prefer references to be listed in a separate section as well. That helps to build a better context of a blog/paper even without reading it thoroughly.
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Mon, 22 Apr 2019 17:29:50 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#2).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2018-12-20-location-awareness.md
1 file changed, 154 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/2
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Greg Solovyev, Grant Henke, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#12).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2019-04-30-location-awareness.md
1 file changed, 380 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/12
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 12
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 10:
(8 comments)
Just a few small comments. I think this is almost good to go.
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md
File _posts/2019-04-16-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@48
PS10, Line 48: leverage the hierarchical structure
: in future releases.
Is there a Jira tracking this?
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@62
PS10, Line 62: (the latter renders the tablet
: server unusable to Kudu clients).
How should users detect this? Does the tablet server fail? Is there a log to look for?
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@73
PS10, Line 73: assigned client's
client's assigned
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@81
PS10, Line 81: follow the following
follow the following just sounds funny because it's two forms for "follow". How about "adhere to the following"?
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@85
PS10, Line 85: as _replica placement policy_
as the replica placement policy
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@87
PS10, Line 87: the
remove "the"
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@103
PS10, Line 103: in future releases that might be done automatically in background
Is thee a jira to track this?
http://gerrit.cloudera.org:8080/#/c/12119/10/_posts/2019-04-16-location-awareness.md@368
PS10, Line 368: References
Now that these are linked inline, are they needed here too?
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 10
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 17 Apr 2019 18:36:55 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 12: Verified+1 Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 12
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 30 Apr 2019 15:40:33 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 1:
(46 comments)
Gerrit gets funny when there's a lot of comments on one file, it seems, so I'm posting this first batch and I will continue reviewing later.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@7
PS1, Line 7: a high-level overview
An overview is always high-level, so remove "high-level".
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@9
PS1, Line 9: principles behind the design decisions
principles of the design
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@11
PS1, Line 11: ways how can be extended and enhanced in the future
potential future enhancements and extensions
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@17
PS1, Line 17: the
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@17
PS1, Line 17: to fit
for
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@20
PS1, Line 20: data
replicas
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@20
PS1, Line 20: Having servers in multiple racks in a Kudu cluster
In a Kudu cluster consisting of multiple servers spread over several racks
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@21
PS1, Line 21: 's data
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@21
PS1, Line 21: replicated
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@21
PS1, Line 21:
in
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@22
PS1, Line 22: becomes
become
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@21
PS1, Line 21: for
: reading and writing
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@22
PS1, Line 22: if any subset of servers
even if all the servers
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@25
PS1, Line 25: In other words, a single rack failure should not lead to data unavailability
: in a Kudu cluster.
Remove (redundant).
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@28
PS1, Line 28: In that sense, a rack is a fault domain (usually, a fault domain
: defined is a set of hardware components that share a single point of failure).
I don't think we need to use this concept very much so I wouldn't include it.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@31
PS1, Line 31: To generalize, replace 'rack' with any other physical aggregation of nodes
Join this paragraph with the previous paragraph.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@31
PS1, Line 31: To generalize
nit: I prefer "More generally", but "To generalize" is ok.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@31
PS1, Line 31: physical
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@32
PS1, Line 32: chassis
Is a chassis a common term for something multiple servers would run on? To me it sounds synonymous with a single physical host, unless the cluster is running in VMs and some VMs share a physical host. If that's the case, I think we ought to say "shared VM host" or some such instead of "chassis".
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@32
PS1, Line 32: site
Does a site differ from a datacenter?
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@33
PS1, Line 33: It might be applicable even to a datacenter
: given that the network latency between datacenters is negligible
: (which, frankly, isn't quite realistic).
Rephrase this more positively: "This even applies to a datacenter if the network latency between datacenters is low."
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@37
PS1, Line 37: At this point, the location awareness feature is a work in progress and may be
: released with Kudu 1.9.0.
I'm not sure what our publishing schedule is, but I think we ought not publish this until the feature is sure to be released with the next release and the next release is coming in the next month or two. In that case, we shouldn't include this sentence, and we should mention the release the feature will first appear in above the fold.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@49
PS1, Line 49: exploiting
to exploit
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@49
PS1, Line 49: for
to establish
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@50
PS1, Line 50: other components of the Hadoop ecosystem (e.g., HDFS)
I think it's just HDFS.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: location
a location
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: leader master
Actually all masters assign a location, but it's the leader master's location that is used in placement decisions, etc.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: for
to
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: Kudu
The
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: upon its registration
when it registers
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@58
PS1, Line 58: being registered
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@58
PS1, Line 58: location
the location
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@59
PS1, Line 59:
the
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@60
PS1, Line 60: for the specified IP address/hostname
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@60
PS1, Line 60: a
the corresponding
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@62
PS1, Line 62: next registration of the tablet server
the tablet server re-registers, which only occurs if the master or tablet server restarts.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@62
PS1, Line 62: The assigned
: location is not reported back to the tablet server since Kudu tablet servers
: are location-agnostic as of now
nit: I would rephrase as "Kudu tablet servers are location agnostic, at least for now, so the assigned location is not reported back to the tablet server.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@64
PS1, Line 64: : masters
. Masters
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@65
PS1, Line 65: internally
Explain how, briefly (maybe by just pointing the reader to the section below).
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@65
PS1, Line 65: provide connected clients with that information as well
Ditto.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@67
PS1, Line 67: Kudu leader master assigns location for a client upon connecting to
: a cluster (i.e. when the client invokes the `ConnectToMaster` RPC).
: The assigned location is sent back to the client along with other
: information in response to `ConnectToMaster` RPC.
I think this is too detailed, but if you want to include it mention the `ConnectToMaster` RPCs above when you first introduce how tablet servers are assigned location when they register.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@73
PS1, Line 73: `--location_mapping_cmd` master's
Swap these two words.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@76
PS1, Line 76:
Does this paragraph belong in the next section? It repeats part of it.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@84
PS1, Line 84: 3 (three)
My preference is to use the word "three", not the symbol "3", but either is OK. Putting both is unnecessary, I think.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@97
PS1, Line 97: By design, keeping the target replication factor for tablets has higher
I'd first restate that re-replication and the placement of new replicas attempts to spread replicas across locations so that the failure of tablet servers in one location does not make tablets unavailable.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@113
PS1, Line 113: is
are
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 1
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 26 Dec 2018 16:49:48 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Greg Solovyev, Grant Henke, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#11).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2019-04-23-location-awareness.md
1 file changed, 380 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/11
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Reviewed-on: http://gerrit.cloudera.org:8080/12119
Reviewed-by: Grant Henke <gr...@apache.org>
Tested-by: Grant Henke <gr...@apache.org>
---
A _posts/2019-04-30-location-awareness.md
1 file changed, 380 insertions(+), 0 deletions(-)
Approvals:
Grant Henke: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: merged
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 13
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12119/3/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
PS3:
The rendered version is available at:
https://gist.github.com/alexeyserbin/c6f03f1d20151d906bf230e9b014dfb7
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 17 Jan 2019 04:35:02 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 4:
(18 comments)
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@50
PS4, Line 50:
the
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@55
PS4, Line 55: location string for the specified IP address/hostname.
: The
Is this meant to be a paragraph break?
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@61
PS4, Line 61: try
to try
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@61
PS4, Line 61: the replica
replicas
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@62
PS4, Line 62: if
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@62
PS4, Line 62: keep
to keep
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@64
PS4, Line 64: as well
Remove, or remove "Also".
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@65
PS4, Line 65: about to connect to the closest tablet server
I'd say "attempt to read from the closest tablet server" since that's all we actually use it for.
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@67
PS4, Line 67: The Kudu leader master assigns location for a client when it connects to a
: cluster. The assigned location string is sent back from the leader master
: to the client along with other cluster-specific information.
This is redundant with the previous paragraph.
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@84
PS4, Line 84:
a
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@92
PS4, Line 92: placement policies
I prefer to say "placement policy" singular since we only have one. Also I think you ought to define the placement policy here (i.e. restate the "use case" from the beginning as the placement policy).
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@94
PS4, Line 94: The automatic re-replication and the initial placement of new replicas both
: attempt to spread replicas across locations so that the failure of tablet
: servers in one location does not make tablets unavailable.
This is basically the placement policy, so I'd just make it clear that "placement policy" is referring to this condition.
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@101
PS4, Line 101: policies
Here and below, like I said, I prefer "policy" since we have just one.
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@139
PS4, Line 139: location
each location
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@142
PS4, Line 142: Examples
This section is really nice. Thanks for adding it.
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@147
PS4, Line 147: squares
nit: rectangles or boxes
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@181
PS4, Line 181: result
resulting
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@188
PS4, Line 188: land
receive
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 24 Jan 2019 18:57:47 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 8:
(28 comments)
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md
File _posts/2019-03-25-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@1
PS8, Line 1: ---
Can you push this to the gh_pages branch in your github fork so a rendered version can be proofed?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@15
PS8, Line 15: <!--TODO(aserbin) rename the file to reflect the date when published -->
Should this be removed?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@19
PS8, Line 19: first cut
initial implementation?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@19
PS8, Line 19: starting 1.9.0
...starting *with the* 1.9.0...
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@20
PS8, Line 20: is built for the following use case:
I am not sure this is a "use case" per se, but instead what the term "location awareness" currently means in Kudu. Maybe say something like:
"In the initial implementation of location awareness in Kudu, when a Kudu cluster consists of multiple servers spread across several racks, Kudu will place the replicas of a tablet in such a way that the tablet stays available even if all the servers in a single rack become unavailable."
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@26
PS8, Line 26: A rack failure might happen because of a failure of a hardware component shared
: among servers in the rack: network switch, power supply, etc.
A rack failure can occur when a hardware component shared
among servers in the rack, such as a network switch or power supply, fails.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@31
PS8, Line 31: network latency between datacenters is low.
This is a good opportunity to explicitly mention that this is why we call the feature location awareness and not rack awareness.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@37
PS8, Line 37: are
: supposed to
should
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@38
PS8, Line 38: physical or cloud-defined hierarchy of the
: deployed cluster
I am not sure I understand what this means in relation to location awareness utility. I suspect it's saying that the components should map to the hierarchical levels of "failure domains".
You could then give a private data center example:
`/data-center-0/rack-09`
And a cloud example:
`/region-0/availability-zone-01`
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@41
PS8, Line 41: However, we want to keep the hierarchy
: there to make it possible to exploit it later
However, we plan to leverage the hierarchical structure in future releases.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@43
PS8, Line 43: compatibility with HDFS
Perhaps this should be moved up and describe a bit more in detail as a design choice? It's useful to know that you can use the same locations as your HDFS nodes, because it's common to deploy Kudu along size HDFS.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@52
PS8, Line 52: etc
What is the "etc"? What else does it use it for?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@55
PS8, Line 55: location string for the specified IP address/hostname.
The script below specifically shows ip-address. How do I use hostname?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@59
PS8, Line 59: tablet server restarts
Is this dependent on `--follower_unavailable_considered_failed_sec`? Or will a "quick" restart cause the location to be reset?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@59
PS8, Line 59: Kudu tablet servers are location
: agnostic, at least for now, so the assigned location is not reported back
: to the tablet server.
Maybe this paragraph would flow better if you moved this part to the bottom. That would make it so you describe how the master uses the location configurations, and then tack on at the end that the tablet servers do not need/use it.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@64
PS8, Line 64: masters provide connected clients
How do they do this?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@62
PS8, Line 62: to try to place replicas evenly across
: locations and to keep tablets available in case all tablet servers in a single
: location fail.
This last part is somewhat duplicated from the Introduction section above. Perhaps it's not needed here.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@75
PS8, Line 75: Essentially, that's about having
This results in...
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@81
PS8, Line 81: The error handling and the input validation are minimalistic. Also, the
: # network topology choice, supportability and capacity planning aspects of
: # this script might be sub-optimal if applied as-is for real-world use cases.
Is there anywhere else anyone can get a "good" production worthy example? If not from us, from who? This leaves the reader with a lot of concerning questions.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@104
PS8, Line 104: echo "ERROR: '$ip_address' is not a valid IPv4 address"
Should errors map to "/other"? How does Kudu handle this script exiting with a non-zero exit code?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@142
PS8, Line 142: The reasoning is simple: with
I try to stay away from saying something is "simple". People have wide levels of experience with distributed systems. Maybe something like:
"It's recommended to have at least three locations defined in a Kudu
cluster so that no location contains a majority of replicas of a tablet."
Then below you can mention the replication factor of 3 in your example.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@151
PS8, Line 151: The location-aware placement policy for tablet replicas in Kudu
This seems more appropriate for earlier sections. When reading the blog post I got the idea that the structure was:
- What it is
- How it works
- How to use it
- Future work
We are now in the "How to use it" part, but this is more about how it works.
Can users configure these policies? Are there more than one?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@162
PS8, Line 162: Automatic re-replication and placement policy
Per my earlier comment, this is also more about "How it works".
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@177
PS8, Line 177: Reinstating location-aware policy in Kudu cluster
I think this is "How to use it" and makes sense here.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@208
PS8, Line 208: Examples
Per my earlier comment, this is also more about "How it works".
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@337
PS8, Line 337: roadmap
What roadmap? Does Apache Kudu have a roadmap?
Maybe we should open jiras and link them for any future work/ideas.
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@342
PS8, Line 342: see [2]
Any reason not to link inline instead of using reference style?
http://gerrit.cloudera.org:8080/#/c/12119/8/_posts/2019-03-25-location-awareness.md@346
PS8, Line 346: [[1]] [Location awareness in Kudu, design document](https://s.apache.org/location-awareness-design)
Can we check this design doc into https://github.com/apache/kudu/tree/master/docs/design-docs and link there?
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 8
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 26 Mar 2019 04:05:31 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 2:
(21 comments)
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@70
PS2, Line 70:
> a
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@98
PS2, Line 98:
> a
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@112
PS2, Line 112: when
> where
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@114
PS2, Line 114: is
> are
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@115
PS2, Line 115: a
> the
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@117
PS2, Line 117:
> the
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@117
PS2, Line 117: kudu
> Either Kudu, or put in backticks `kudu`.
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@121
PS2, Line 121: very
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@121
PS2, Line 121: At
> In
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@122
PS2, Line 122: policies
> policy
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@122
PS2, Line 122: re-establishes
> reestablish
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@124
PS2, Line 124: continue to the cross-location rebalancing.
> Add "phase" at the end.
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@126
PS2, Line 126: Second phase is cross-location rebalancing, i.e. moving tablet replicas between
> "The second phase"
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@127
PS2, Line 127: location
> locations
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@128
PS2, Line 128: loads
> What is the load of a location?
I added a sentence about that.
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@128
PS2, Line 128: equalizing loads of locations throughout the cluster. Use the
> the
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@132
PS2, Line 132: Third phase is intra-location rebalancing, i.e. balance tablet replica
> The
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@132
PS2, Line 132: replica
> replicas
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@133
PS2, Line 133: distribution
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@133
PS2, Line 133: such
> the
Done
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@135
PS2, Line 135:
> A very quick example showing the moves for each of the three phases would b
Done.
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 17 Jan 2019 04:33:11 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Greg Solovyev (Code Review)" <ge...@cloudera.org>.
Greg Solovyev has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 3:
It would be good to provide a working code sample for location mapping executable in order to demonstrate input/output expectations. Even though this seems trivial having a code sample will eliminate potential ambiguity.
It would also be good to mention in which version of Kudu this is available.
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 18 Jan 2019 16:22:27 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12119/9//COMMIT_MSG
Commit Message:
PS9:
The rendered version of this blogpost is available at https://github.com/alexeyserbin/kudu/blob/gh-pages/_posts/2019-03-25-location-awareness.md
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 9
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 26 Mar 2019 18:07:27 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 3:
(1 comment)
> (1 comment)
>
> > We have this script which we use for testing:
> > https://github.com/apache/kudu/blob/master/src/kudu/scripts/assign-location.py
>
> While that script is awesome for testing it's a really bad example
> for users because it doesn't actually assign locations based on the
> network topology. We should make a small example that assigns
> locations based on common subnet or something like that.
SGTM. I agree it's better to have something that closer reflects the reality. After some consideration, I think out test scripts might be really confusing for people trying to understand what it's all about.
>
> > Maybe, I should put some TODO in the text as a reminder?
>
> Good idea. Just make it a comment.
Done.
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@37
PS2, Line 37: `/dc-0/rack-09/host-01`
> This isn't a good example location because it would place the host in its o
Done
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Sat, 19 Jan 2019 04:48:21 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 7: Code-Review+2
Don't forget to update the date like Todd said.
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 7
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 31 Jan 2019 18:30:34 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 3:
(1 comment)
> We have this script which we use for testing:
> https://github.com/apache/kudu/blob/master/src/kudu/scripts/assign-location.py
While that script is awesome for testing it's a really bad example for users because it doesn't actually assign locations based on the network topology. We should make a small example that assigns locations based on common subnet or something like that.
> Maybe, I should put some TODO in the text as a reminder?
Good idea. Just make it a comment.
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@37
PS2, Line 37: `/dc-0/rack-09/host-01`
This isn't a good example location because it would place the host in its own location, which probably isn't what a user wants. I think `/dc-0/rack-09` is better.
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Sat, 19 Jan 2019 00:58:32 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 5:
(18 comments)
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@50
PS4, Line 50:
> the
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@55
PS4, Line 55: location string for the specified IP address/hostname.
:
> Is this meant to be a paragraph break?
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@61
PS4, Line 61: as
> to try
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@61
PS4, Line 61: ation infor
> replicas
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@62
PS4, Line 62: o
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@62
PS4, Line 62: a pl
> to keep
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@64
PS4, Line 64: de conn
> Remove, or remove "Also".
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@65
PS4, Line 65: nt's location, so the clients can make inform
> I'd say "attempt to read from the closest tablet server" since that's all w
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@67
PS4, Line 67:
: How to make a Kudu cluster location-aware
: ==========
> This is redundant with the previous paragraph.
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@84
PS4, Line 84:
> a
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@92
PS4, Line 92: ke tablets unavail
> I prefer to say "placement policy" singular since we only have one. Also I
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@94
PS4, Line 94: That's referred to as _replica placement policy_ or just _placement policy_.
: In Kudu, both the initial placement of tablet replicas and the automatic
: re-replication are governed by the that policy.
> This is basically the placement policy, so I'd just make it clear that "pla
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@101
PS4, Line 101: s,
> Here and below, like I said, I prefer "policy" since we have just one.
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@139
PS4, Line 139:
> each location
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@142
PS4, Line 142: own. Use
> This section is really nice. Thanks for adding it.
I added it while waiting for my PHX --> SJC flight in the Phoenix airport :)
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@147
PS4, Line 147:
> nit: rectangles or boxes
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@181
PS4, Line 181:
> resulting
Done
http://gerrit.cloudera.org:8080/#/c/12119/4/_posts/2018-12-20-location-awareness.md@188
PS4, Line 188: o lo
> receive
Done
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 25 Jan 2019 22:59:00 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 2:
(46 comments)
Thank you for the thorough review!
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@7
PS1, Line 7: an overview
> An overview is always high-level, so remove "high-level".
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@9
PS1, Line 9: principles of the design
> principles of the design
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@11
PS1, Line 11: potential future enhancements and extensions
> potential future enhancements and extensions
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@17
PS1, Line 17: the fo
> for
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@17
PS1, Line 17: loc
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@20
PS1, Line 20: In a Kudu cluster consisting of multiple servers s
> In a Kudu cluster consisting of multiple servers spread over several racks
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@20
PS1, Line 20: ever
> replicas
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@21
PS1, Line 21: that t
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@21
PS1, Line 21: s
> in
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@21
PS1, Line 21: place the
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@21
PS1, Line 21: vailable
: even if all the ser
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@22
PS1, Line 22:
> become
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@22
PS1, Line 22: ers in a single rack bec
> even if all the servers
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@25
PS1, Line 25: among servers in the rack: network switch, power supply, etc. More generally,
: replace 'rack' wit
> Remove (redundant).
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@28
PS1, Line 28: lies to a datacenter if the network latency between datacenters
: is low.
> I don't think we need to use this concept very much so I wouldn't include i
OK. I thought this might look easier to digest for people who know what fault domain is, but I agree we don't use the concept of fault domain in this post, so it's not necessary to include.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@31
PS1, Line 31:
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@31
PS1, Line 31: Locations in
> nit: I prefer "More generally", but "To generalize" is ok.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@31
PS1, Line 31: Locations in Kudu
> Join this paragraph with the previous paragraph.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@32
PS1, Line 32: ===
> Is a chassis a common term for something multiple servers would run on? To
In big server farms, they sometimes have aggregates of physical hosts/nodes (sometimes hot-pluggable) which share common power, e.g.: https://ark.intel.com/products/81878/Intel-Server-Chassis-H2216XXKR2 (check the 'Product images' page to get a better idea how it looks like).
Those are called 'chassis'.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@32
PS1, Line 32:
> Does a site differ from a datacenter?
Site might mean a sub-unit in a datacenter (e.g., a server room in a big datacenter).
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@33
PS1, Line 33: a string that begins with a slash (`/`) and
: consists of slash-separated tokens each of which contains only characters from
: the set `[a-zA-Z0-9_-.]`. The components
> Rephrase this more positively: "This even applies to a datacenter if the ne
I like that positive attitude :)
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@37
PS1, Line 37: deployed cluster, e.g. `/dc-0/rack-09/host-01`. As of now, Kudu does not
: exploit the hierarchical
> I'm not sure what our publishing schedule is, but I think we ought not publ
OK, I removed this sentence and we can add information on the version once the release is cut.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@49
PS1, Line 49: registers
> to exploit
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@49
PS1, Line 49: d b
> to establish
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@50
PS1, Line 50: r is used to make tablet replica placement decisions,
> I think it's just HDFS.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: tic, at least
> Actually all masters assign a location, but it's the leader master's locati
OK, I rephrased this accordingly.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: agno
> The
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: ssi
> to
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: so the
> a location
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@57
PS1, Line 57: not reported back
> when it registers
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@58
PS1, Line 58: ormation interna
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@58
PS1, Line 58: use the
> the location
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@59
PS1, Line 59:
> the
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@60
PS1, Line 60: in case if all tablet servers in a si
> Remove.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@60
PS1, Line 60: t
> the corresponding
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@62
PS1, Line 62: ocation as well, so the clients can ma
> the tablet server re-registers, which only occurs if the master or tablet s
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@62
PS1, Line 62: informed
: decisions when they about to connect to the closest tablet server.
:
> nit: I would rephrase as "Kudu tablet servers are location agnostic, at lea
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@64
PS1, Line 64:
> . Masters
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@65
PS1, Line 65: master assigns location for a client when it connects
> Ditto.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@65
PS1, Line 65: The Kudu l
> Explain how, briefly (maybe by just pointing the reader to the section belo
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@67
PS1, Line 67: to the client along with other cluster-specific information.
:
:
: How to make Kudu cluster location-aware
> I think this is too detailed, but if you want to include it mention the `Co
I removed the details.
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@73
PS1, Line 73: ion_mapping_cmd` flag for Kudu ma
> Swap these two words.
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@76
PS1, Line 76: mappings stay the same regardless of the node where the location assignment
> Does this paragraph belong in the next section? It repeats part of it.
Yup, indeed -- that's some variation of the paragraph below. Good catch!
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@84
PS1, Line 84: r in a si
> My preference is to use the word "three", not the symbol "3", but either is
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@97
PS1, Line 97: priority than conforming to the replica placement policies. In other words,
> I'd first restate that re-replication and the placement of new replicas att
Done
http://gerrit.cloudera.org:8080/#/c/12119/1/_posts/2018-12-20-location-awareness.md@113
PS1, Line 113: s
> are
Done
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Fri, 28 Dec 2018 23:26:03 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 2:
(21 comments)
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@70
PS2, Line 70:
a
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@98
PS2, Line 98:
a
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@112
PS2, Line 112: when
where
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@114
PS2, Line 114: is
are
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@115
PS2, Line 115: a
the
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@117
PS2, Line 117:
the
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@117
PS2, Line 117: kudu
Either Kudu, or put in backticks `kudu`.
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@121
PS2, Line 121: very
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@121
PS2, Line 121: At
In
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@122
PS2, Line 122: policies
policy
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@122
PS2, Line 122: re-establishes
reestablish
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@124
PS2, Line 124: continue to the cross-location rebalancing.
Add "phase" at the end.
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@126
PS2, Line 126: Second phase is cross-location rebalancing, i.e. moving tablet replicas between
"The second phase"
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@127
PS2, Line 127: location
locations
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@128
PS2, Line 128: equalizing loads of locations throughout the cluster. Use the
the
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@128
PS2, Line 128: loads
What is the load of a location?
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@132
PS2, Line 132: replica
replicas
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@132
PS2, Line 132: Third phase is intra-location rebalancing, i.e. balance tablet replica
The
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@133
PS2, Line 133: such
the
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@133
PS2, Line 133: distribution
Remove.
http://gerrit.cloudera.org:8080/#/c/12119/2/_posts/2018-12-20-location-awareness.md@135
PS2, Line 135:
A very quick example showing the moves for each of the three phases would be good. It would illustrate the three phases and the role of "load" in cross-location rebalancing.
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 2
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 08 Jan 2019 21:40:44 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Greg Solovyev, Grant Henke, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#10).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2019-04-16-location-awareness.md
1 file changed, 372 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/10
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 10
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Greg Solovyev, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#6).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2018-12-20-location-awareness.md
1 file changed, 348 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/6
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#3).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2018-12-20-location-awareness.md
1 file changed, 280 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/3
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Greg Solovyev, Grant Henke, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#9).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2019-03-25-location-awareness.md
1 file changed, 374 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/9
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 9
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/12119/6/_posts/2018-12-20-location-awareness.md
File _posts/2018-12-20-location-awareness.md:
PS6:
The rendered version is at:
https://gist.github.com/alexeyserbin/c6f03f1d20151d906bf230e9b014dfb7
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 6
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 29 Jan 2019 19:30:34 +0000
Gerrit-HasComments: Yes
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Mon, 28 Jan 2019 22:25:40 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 3:
> It would be good to provide a working code sample for location
> mapping executable in order to demonstrate input/output
> expectations. Even though this seems trivial having a code sample
> will eliminate potential ambiguity.
We have this script which we use for testing:
https://github.com/apache/kudu/blob/master/src/kudu/scripts/assign-location.py
Do you think it's good enough or the idea is to have something as a snippet in the blog post itself?
>
> It would also be good to mention in which version of Kudu this is
> available.
Yep, we discussed that in the prior comments with Will. The idea is to put the version in the very top as soon as 1.9 get released. Maybe, I should put some TODO in the text as a reminder?
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 3
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Sat, 19 Jan 2019 00:35:34 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 9:
Now that the design doc is merged the link can be updated.
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 9
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Mon, 15 Apr 2019 14:17:08 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Grant Henke (Code Review)" <ge...@cloudera.org>.
Grant Henke has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 11: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 11
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Mon, 22 Apr 2019 19:48:19 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 5
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Mon, 28 Jan 2019 22:25:47 +0000
Gerrit-HasComments: No
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Greg Solovyev,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/12119
to look at the new patch set (#4).
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
[blog] a blogpost about location awareness in Kudu
Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
---
A _posts/2018-12-20-location-awareness.md
1 file changed, 282 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/19/12119/4
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 4
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR](gh-pages) [blog] a blogpost about location awareness in Kudu
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/12119 )
Change subject: [blog] a blogpost about location awareness in Kudu
......................................................................
Patch Set 10:
> Patch Set 9:
>
> Now that the design doc is merged the link can be updated.
Yes, indeed. Thank you for kind reminder!
--
To view, visit http://gerrit.cloudera.org:8080/12119
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: gh-pages
Gerrit-MessageType: comment
Gerrit-Change-Id: I10b30a80d5661fb889a11285b8118cdea1a87cd2
Gerrit-Change-Number: 12119
Gerrit-PatchSet: 10
Gerrit-Owner: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Grant Henke <gr...@apache.org>
Gerrit-Reviewer: Greg Solovyev <gs...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Mon, 15 Apr 2019 22:22:21 +0000
Gerrit-HasComments: No