You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Will Berkeley (Code Review)" <ge...@cloudera.org> on 2018/05/01 18:37:37 UTC
[kudu-CR] Add warnings and docs on setting block cache capacity mb too high
Will Berkeley has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10266
Change subject: Add warnings and docs on setting block_cache_capacity_mb too high
......................................................................
Add warnings and docs on setting block_cache_capacity_mb too high
Raising --block_cache_capacity_mb can improve Kudu's performance, but
raising it too high (above the memory pressure threshold, which is
--memory_pressure_percentage of the hard memory limit) causes
;wq
constant flushing even if write throughput is low. This patch adds
some log messages warning about the potential problems with
setting block_cache_capacity_mb too high, as well as docs.
I chose 50% as the max recommended block cache size as a
percentage of the memory pressure threshold somewhat arbitrarily.
It seems safe, but also leaves room for allocating a large
block cache.
There's no automated tests but I did verify the expected messages
appeared in the logs when --block_cache_capacity_mb is too
high.
Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
---
M docs/troubleshooting.adoc
M src/kudu/cfile/block_cache.cc
M src/kudu/util/process_memory.cc
M src/kudu/util/process_memory.h
4 files changed, 40 insertions(+), 0 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/1
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10266
to look at the new patch set (#7).
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Add validation and docs on setting block_cache_capacity_mb too high
Raising --block_cache_capacity_mb can improve Kudu's performance, but
raising it too high (above the memory pressure threshold, which is
--memory_pressure_percentage of the hard memory limit) causes
constant flushing even if write throughput is low. This patch adds
validation of the capacity using a group flag validator, and does
not permit Kudu to start if the block cache size can can cause
Kudu to get stuck in aggressive flushing mode. It warns if the
capacity is large enough (> 50%) of the pressure threshold.
I chose 50% as the max recommended block cache size as a
percentage of the memory pressure threshold somewhat arbitrarily.
It seems safe, but also leaves room for allocating a large
block cache.
There's no automated tests but I did verify the expected messages
appeared in the logs and the server fails to start when
--block_cache_capacity_mb is too high.
Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
---
M docs/troubleshooting.adoc
M src/kudu/cfile/block_cache.cc
M src/kudu/integration-tests/client-stress-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/util/process_memory.cc
M src/kudu/util/process_memory.h
6 files changed, 76 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/7
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 7
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10266
to look at the new patch set (#4).
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Add validation and docs on setting block_cache_capacity_mb too high
Raising --block_cache_capacity_mb can improve Kudu's performance, but
raising it too high (above the memory pressure threshold, which is
--memory_pressure_percentage of the hard memory limit) causes
constant flushing even if write throughput is low. This patch adds
validation of the capacity using a group flag validator, and does
not permit Kudu to start if the block cache size can can cause
Kudu to get stuck in aggressive flushing mode. It warns if the
capacity is large enough (> 50%) of the pressure threshold.
I chose 50% as the max recommended block cache size as a
percentage of the memory pressure threshold somewhat arbitrarily.
It seems safe, but also leaves room for allocating a large
block cache.
There's no automated tests but I did verify the expected messages
appeared in the logs and the server fails to start when
--block_cache_capacity_mb is too high.
Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
---
M docs/troubleshooting.adoc
M src/kudu/cfile/block_cache.cc
M src/kudu/integration-tests/client-stress-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/util/process_memory.cc
M src/kudu/util/process_memory.h
6 files changed, 63 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/4
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has removed a vote on this change.
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Removed Verified-1 by Kudu Jenkins (120)
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10266
to look at the new patch set (#3).
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Add validation and docs on setting block_cache_capacity_mb too high
Raising --block_cache_capacity_mb can improve Kudu's performance, but
raising it too high (above the memory pressure threshold, which is
--memory_pressure_percentage of the hard memory limit) causes
constant flushing even if write throughput is low. This patch adds
validation of the capacity using a group flag validator, and does
not permit Kudu to start if the block cache size can can cause
Kudu to get stuck in aggressive flushing mode. It warns if the
capacity is large enough (> 50%) of the pressure threshold.
I chose 50% as the max recommended block cache size as a
percentage of the memory pressure threshold somewhat arbitrarily.
It seems safe, but also leaves room for allocating a large
block cache.
There's no automated tests but I did verify the expected messages
appeared in the logs and the server fails to start when
--block_cache_capacity_mb is too high.
Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
---
M docs/troubleshooting.adoc
M src/kudu/cfile/block_cache.cc
M src/kudu/util/process_memory.cc
M src/kudu/util/process_memory.h
4 files changed, 56 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/3
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 3
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Adar Dembo, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10266
to look at the new patch set (#6).
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Add validation and docs on setting block_cache_capacity_mb too high
Raising --block_cache_capacity_mb can improve Kudu's performance, but
raising it too high (above the memory pressure threshold, which is
--memory_pressure_percentage of the hard memory limit) causes
constant flushing even if write throughput is low. This patch adds
validation of the capacity using a group flag validator, and does
not permit Kudu to start if the block cache size can can cause
Kudu to get stuck in aggressive flushing mode. It warns if the
capacity is large enough (> 50%) of the pressure threshold.
I chose 50% as the max recommended block cache size as a
percentage of the memory pressure threshold somewhat arbitrarily.
It seems safe, but also leaves room for allocating a large
block cache.
There's no automated tests but I did verify the expected messages
appeared in the logs and the server fails to start when
--block_cache_capacity_mb is too high.
Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
---
M docs/troubleshooting.adoc
M src/kudu/cfile/block_cache.cc
M src/kudu/integration-tests/client-stress-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/util/process_memory.cc
M src/kudu/util/process_memory.h
6 files changed, 77 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/6
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 6
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10266/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10266/2//COMMIT_MSG@12
PS2, Line 12: ;wq
> typo
Done
http://gerrit.cloudera.org:8080/#/c/10266/2/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/2/src/kudu/cfile/block_cache.cc@84
PS2, Line 84: int64 mpt = process_memory::MemoryPressureThreshold();
: if (capacity > mpt) {
: LOG(ERROR) << "Block cache capacity exceeds the memory pressure threshold "
: "(" << capacity << " bytes vs. " << mpt << " bytes). This "
: "will cause instability and harmful flushing behavior. "
: "Consider lowering --block_cache_capacity_mb or raising "
: "--memory_limit_hard_bytes.";
: } else if (capacity > mpt / 2) {
: LOG(WARNING) << "Block cache capacity exceeds 50% of the memory pressure threshold "
: "(" << capacity << " bytes vs. 50% of " << mpt << " bytes). "
: " This may cause performance problems. Consider lowering "
: "--block_cache_capacity_mb or raising --memory_limit_hard_bytes.";
: }
> Is it possible to perform the verification of these parameters in GROUP_FLA
Done. I also decided to upgrade the first condition to be fatal, since the behavior of Kudu in that situation can get pretty bad (KUDU-2318).
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 05:50:19 +0000
Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 4: Verified+1
TSAN failure is 2059.
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 18:17:27 +0000
Gerrit-HasComments: No
[kudu-CR] Add warnings and docs on setting block cache capacity mb too high
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add warnings and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10266/2//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10266/2//COMMIT_MSG@12
PS2, Line 12: ;wq
typo
http://gerrit.cloudera.org:8080/#/c/10266/2/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/2/src/kudu/cfile/block_cache.cc@84
PS2, Line 84: int64 mpt = process_memory::MemoryPressureThreshold();
: if (capacity > mpt) {
: LOG(ERROR) << "Block cache capacity exceeds the memory pressure threshold "
: "(" << capacity << " bytes vs. " << mpt << " bytes). This "
: "will cause instability and harmful flushing behavior. "
: "Consider lowering --block_cache_capacity_mb or raising "
: "--memory_limit_hard_bytes.";
: } else if (capacity > mpt / 2) {
: LOG(WARNING) << "Block cache capacity exceeds 50% of the memory pressure threshold "
: "(" << capacity << " bytes vs. 50% of " << mpt << " bytes). "
: " This may cause performance problems. Consider lowering "
: "--block_cache_capacity_mb or raising --memory_limit_hard_bytes.";
: }
Is it possible to perform the verification of these parameters in GROUP_FLAG_VALIDATOR for the --block_cache_capacity_mb flag? Then it would be possible to modify the behavior and refuse to start (if no appropriate --force_xxx flag specified) for the case of capacity > mpt.
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 01 May 2018 20:53:45 +0000
Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 5:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@80
PS5, Line 80:
nit: extra space
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc
File src/kudu/integration-tests/client-stress-test.cc:
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc@241
PS5, Line 241: // The block cache capacity needs to be disabled. See the group validator
> Hmm, this just disables the block cache outright, correct? Reword to say th
An alternative approach might be using --force_block_cache_capacity (since it's been introduced already) instead of disabling the block cache at all?
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc
File src/kudu/integration-tests/raft_consensus-itest.cc:
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc@1803
PS5, Line 1803: // We also need to limit the block cache capacity in tandem. See the group
> Again, isn't this outright disabling?
Maybe, since the --force_block_cache_capacity is introduced, it would be possible to use it here and don't disable the block cache at all?
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc
File src/kudu/util/process_memory.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc@a182
PS4, Line 182:
:
:
:
:
:
:
:
> It wasn't printed before this change AFAICT, and now the validator makes it
Ah, I see -- that makes sense.
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 20:23:51 +0000
Gerrit-HasComments: Yes
[kudu-CR] Add warnings and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10266
to look at the new patch set (#2).
Change subject: Add warnings and docs on setting block_cache_capacity_mb too high
......................................................................
Add warnings and docs on setting block_cache_capacity_mb too high
Raising --block_cache_capacity_mb can improve Kudu's performance, but
raising it too high (above the memory pressure threshold, which is
--memory_pressure_percentage of the hard memory limit) causes
;wq
constant flushing even if write throughput is low. This patch adds
some log messages warning about the potential problems with
setting block_cache_capacity_mb too high, as well as docs.
I chose 50% as the max recommended block cache size as a
percentage of the memory pressure threshold somewhat arbitrarily.
It seems safe, but also leaves room for allocating a large
block cache.
There's no automated tests but I did verify the expected messages
appeared in the logs when --block_cache_capacity_mb is too
high.
Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
---
M docs/troubleshooting.adoc
M src/kudu/cfile/block_cache.cc
M src/kudu/util/process_memory.cc
M src/kudu/util/process_memory.h
4 files changed, 43 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/2
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 2
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Hello Alexey Serbin, Kudu Jenkins, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10266
to look at the new patch set (#5).
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Add validation and docs on setting block_cache_capacity_mb too high
Raising --block_cache_capacity_mb can improve Kudu's performance, but
raising it too high (above the memory pressure threshold, which is
--memory_pressure_percentage of the hard memory limit) causes
constant flushing even if write throughput is low. This patch adds
validation of the capacity using a group flag validator, and does
not permit Kudu to start if the block cache size can can cause
Kudu to get stuck in aggressive flushing mode. It warns if the
capacity is large enough (> 50%) of the pressure threshold.
I chose 50% as the max recommended block cache size as a
percentage of the memory pressure threshold somewhat arbitrarily.
It seems safe, but also leaves room for allocating a large
block cache.
There's no automated tests but I did verify the expected messages
appeared in the logs and the server fails to start when
--block_cache_capacity_mb is too high.
Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
---
M docs/troubleshooting.adoc
M src/kudu/cfile/block_cache.cc
M src/kudu/integration-tests/client-stress-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/util/process_memory.cc
M src/kudu/util/process_memory.h
6 files changed, 70 insertions(+), 10 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/66/10266/5
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 6
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 20:55:04 +0000
Gerrit-HasComments: No
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 5:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@41
PS5, Line 41: DEFINE_bool(force_block_cache_capacity_mb, false,
> Nit: the _mb suffix makes me think this is a megabyte quantity.
Done
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@77
PS5, Line 77: LOG(ERROR) << "Block cache capacity exceeds the memory pressure threshold "
> Nit: use strings::Substitute() here and below, makes the text a little more
Done
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@80
PS5, Line 80:
> nit: extra space
Done
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@94
PS5, Line 94: // Group validation is necessary since the validator depends on
: // --memory_limit_hard_bytes and other memory flags.
> FWIW, I think this is implied by the content of the validator.
Done
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc
File src/kudu/integration-tests/client-stress-test.cc:
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc@241
PS5, Line 241: // The block cache capacity needs to be disabled. See the group validator
> Hmm, this just disables the block cache outright, correct? Reword to say th
Done
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc@241
PS5, Line 241: // The block cache capacity needs to be disabled. See the group validator
> An alternative approach might be using --force_block_cache_capacity (since
Done
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc
File src/kudu/integration-tests/raft_consensus-itest.cc:
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc@1803
PS5, Line 1803: // We also need to limit the block cache capacity in tandem. See the group
> Maybe, since the --force_block_cache_capacity is introduced, it would be po
Done
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc@1803
PS5, Line 1803: // We also need to limit the block cache capacity in tandem. See the group
> Again, isn't this outright disabling?
Done
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 20:37:52 +0000
Gerrit-HasComments: Yes
[kudu-CR] Add warnings and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add warnings and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/10266/1/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/1/src/kudu/cfile/block_cache.cc@84
PS1, Line 84: ERROR
FATAL potentially? The effects could be pretty bad.
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 1
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 01 May 2018 18:38:51 +0000
Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 5:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@41
PS5, Line 41: DEFINE_bool(force_block_cache_capacity_mb, false,
Nit: the _mb suffix makes me think this is a megabyte quantity.
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@77
PS5, Line 77: LOG(ERROR) << "Block cache capacity exceeds the memory pressure threshold "
Nit: use strings::Substitute() here and below, makes the text a little more clear IMHO.
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/cfile/block_cache.cc@94
PS5, Line 94: // Group validation is necessary since the validator depends on
: // --memory_limit_hard_bytes and other memory flags.
FWIW, I think this is implied by the content of the validator.
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc
File src/kudu/integration-tests/client-stress-test.cc:
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/client-stress-test.cc@241
PS5, Line 241: // The block cache capacity needs to be disabled. See the group validator
Hmm, this just disables the block cache outright, correct? Reword to say that?
Also, please add a few more words about _why_ it needs to be disabled.
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc
File src/kudu/integration-tests/raft_consensus-itest.cc:
http://gerrit.cloudera.org:8080/#/c/10266/5/src/kudu/integration-tests/raft_consensus-itest.cc@1803
PS5, Line 1803: // We also need to limit the block cache capacity in tandem. See the group
Again, isn't this outright disabling?
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 5
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 18:49:58 +0000
Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@67
PS4, Line 67: int64
int64_t ?
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@72
PS4, Line 72:
nit: extra space
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@72
PS4, Line 72:
: return false;
Do you think it's worth to provide an ability to override this and run with those pernicious effects if a special --force_xxx was specified?
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@77
PS4, Line 77:
nit: extra space
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/client-stress-test.cc
File src/kudu/integration-tests/client-stress-test.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/client-stress-test.cc@241
PS4, Line 241: // The block cache capacity needs to be disabled. See the group validator
: // in cfile/block_cache.cc.
: opts.extra_tserver_flags.push_back("--block_cache_capacity_mb=0");
Does it induce some crucial changes in the behavior of the system while running this test?
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/raft_consensus-itest.cc
File src/kudu/integration-tests/raft_consensus-itest.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/raft_consensus-itest.cc@1810
PS4, Line 1810: "--block_cache_capacity_mb=0",
Why not just move one "--block_cache_capacity_mb=0" out of the #ifdefs ?
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc
File src/kudu/util/process_memory.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc@a182
PS4, Line 182:
:
:
:
:
:
:
:
Wasn't it useful at all?
In other words, do we print that info from somewhere else?
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 18:20:36 +0000
Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 7
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 22:09:56 +0000
Gerrit-HasComments: No
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 6: Code-Review+2
LGTM; it's worth checking with Adar as well before submitting.
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 6
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 20:54:29 +0000
Gerrit-HasComments: No
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Patch Set 4:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc
File src/kudu/cfile/block_cache.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@67
PS4, Line 67: int64
> int64_t ?
Done
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@72
PS4, Line 72:
> nit: extra space
Done
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@72
PS4, Line 72:
: return false;
> Do you think it's worth to provide an ability to override this and run with
Sure, as an unsafe flag, if you wanted to see the effects or were working on a patch to mitigate them. Done.
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/cfile/block_cache.cc@77
PS4, Line 77:
> nit: extra space
Done
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/client-stress-test.cc
File src/kudu/integration-tests/client-stress-test.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/client-stress-test.cc@241
PS4, Line 241: // The block cache capacity needs to be disabled. See the group validator
: // in cfile/block_cache.cc.
: opts.extra_tserver_flags.push_back("--block_cache_capacity_mb=0");
> Does it induce some crucial changes in the behavior of the system while run
Don't think so. Since the soft limit percentage is 0, the servers are always under memory pressure anyway.
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/raft_consensus-itest.cc
File src/kudu/integration-tests/raft_consensus-itest.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/integration-tests/raft_consensus-itest.cc@1810
PS4, Line 1810: "--block_cache_capacity_mb=0",
> Why not just move one "--block_cache_capacity_mb=0" out of the #ifdefs ?
Done
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc
File src/kudu/util/process_memory.cc:
http://gerrit.cloudera.org:8080/#/c/10266/4/src/kudu/util/process_memory.cc@a182
PS4, Line 182:
:
:
:
:
:
:
:
> Wasn't it useful at all?
It wasn't printed before this change AFAICT, and now the validator makes it get printed before logging is initialized, which is annoying. The info is available from flag values (and arithmetic).
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 4
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Thu, 03 May 2018 18:31:14 +0000
Gerrit-HasComments: Yes
[kudu-CR] Add validation and docs on setting block cache capacity mb too high
Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10266 )
Change subject: Add validation and docs on setting block_cache_capacity_mb too high
......................................................................
Add validation and docs on setting block_cache_capacity_mb too high
Raising --block_cache_capacity_mb can improve Kudu's performance, but
raising it too high (above the memory pressure threshold, which is
--memory_pressure_percentage of the hard memory limit) causes
constant flushing even if write throughput is low. This patch adds
validation of the capacity using a group flag validator, and does
not permit Kudu to start if the block cache size can can cause
Kudu to get stuck in aggressive flushing mode. It warns if the
capacity is large enough (> 50%) of the pressure threshold.
I chose 50% as the max recommended block cache size as a
percentage of the memory pressure threshold somewhat arbitrarily.
It seems safe, but also leaves room for allocating a large
block cache.
There's no automated tests but I did verify the expected messages
appeared in the logs and the server fails to start when
--block_cache_capacity_mb is too high.
Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Reviewed-on: http://gerrit.cloudera.org:8080/10266
Reviewed-by: Alexey Serbin <as...@cloudera.com>
Tested-by: Kudu Jenkins
---
M docs/troubleshooting.adoc
M src/kudu/cfile/block_cache.cc
M src/kudu/integration-tests/client-stress-test.cc
M src/kudu/integration-tests/raft_consensus-itest.cc
M src/kudu/util/process_memory.cc
M src/kudu/util/process_memory.h
6 files changed, 76 insertions(+), 10 deletions(-)
Approvals:
Alexey Serbin: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/10266
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia99fa95c578235beb3a1c72a33a22b4d8ff4800c
Gerrit-Change-Number: 10266
Gerrit-PatchSet: 8
Gerrit-Owner: Will Berkeley <wd...@gmail.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <as...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>