You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2018/02/08 03:55:54 UTC

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection 

Hello Will Berkeley, Mike Percy,

I'd like you to do a code review. Please visit

    http://gerrit.cloudera.org:8080/9254

to review the following change.


Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................

KUDU-2291 (part 3): use futex to speed up stack collection

Previously the thread which requests a stack trace would poll on an
atomic variable waiting for the stack to be collected. Of course
spinning is too CPU-hungry so it would actually sleep for 10ms in
each iteration. This limited the performance of stack collection to
about 100/second, which is problematic for the /stacks web page on a
server that may have thousands of threads.

This switches to using the futex system call to allow the dumper thread
to sleep and the dumpee thread to wake it back up when the results are
ready.

Why do we use the low level futex and not something like a condition
variable or CountdownLatch? The issue is that the dumpee thread is
running in a signal handler context, and most things are not safe to do
in that context. Particularly, mutexes may block, which is very naughty,
and condition variables and latches are currently implemented on top of
mutexes. pthread semaphores are another option, but those don't support
proper timeout semantics, which is a feature we use in this use case.

Whether the futex syscall itself is async-signal-safe is up for some debate.
Clearly the "wait" mode is not, since it may block, but is the "wake" mode
that we use here allowed? Technically I don't think so. In practice,
the pthread sem_post() API _is_ documented to be async-signal-safe and
I verified that in libc source it's implemented using futex to wake the
waiter. So, if we are doing something illegal, at least we're doing
the same illegal thing as libc, so we're very unlikely to ever break.

Obviously futex is not portable to macOS, but this code was already
Linux-specific so we haven't lost much.

This speeds up the benchmark for stack trace collection (without symbolization)
by 585x:

Before:
  I0207 19:33:20.960477 30799 debug-util-test.cc:175] Throughput: 99 dumps/second (symbolize=0)
  I0207 19:33:21.963946 30799 debug-util-test.cc:175] Throughput: 84 dumps/second (symbolize=1)

After:
  I0207 19:38:32.505213 32039 debug-util-test.cc:175] Throughput: 57916 dumps/second (symbolize=0)
  I0207 19:38:33.506417 32039 debug-util-test.cc:175] Throughput: 595 dumps/second (symbolize=1)

Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
---
M src/kudu/util/debug-util.cc
1 file changed, 63 insertions(+), 15 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/9254/1
-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 1
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Mike Percy, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/9254

to look at the new patch set (#6).

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................

KUDU-2291 (part 3): use futex to speed up stack collection

Previously the thread which requests a stack trace would poll on an
atomic variable waiting for the stack to be collected. Of course
spinning is too CPU-hungry so it would actually sleep for 10ms in
each iteration. This limited the performance of stack collection to
about 100/second, which is problematic for the /stacks web page on a
server that may have thousands of threads.

This switches to using the futex system call to allow the dumper thread
to sleep and the dumpee thread to wake it back up when the results are
ready.

Why do we use the low level futex and not something like a condition
variable or CountdownLatch? The issue is that the dumpee thread is
running in a signal handler context, and most things are not safe to do
in that context. Particularly, mutexes may block, which is very naughty,
and condition variables and latches are currently implemented on top of
mutexes. pthread semaphores are another option, but those don't support
proper timeout semantics, which is a feature we use in this use case.

Whether the futex syscall itself is async-signal-safe is up for some debate.
Clearly the "wait" mode is not, since it may block, but is the "wake" mode
that we use here allowed? Technically I don't think so. In practice,
the pthread sem_post() API _is_ documented to be async-signal-safe and
I verified that in libc source it's implemented using futex to wake the
waiter. So, if we are doing something illegal, at least we're doing
the same illegal thing as libc, so we're very unlikely to ever break.

Obviously futex is not portable to macOS, but this code was already
Linux-specific so we haven't lost much.

This speeds up the benchmark for stack trace collection (without symbolization)
by 585x:

Before:
  I0207 19:33:20.960477 30799 debug-util-test.cc:175] Throughput: 99 dumps/second (symbolize=0)
  I0207 19:33:21.963946 30799 debug-util-test.cc:175] Throughput: 84 dumps/second (symbolize=1)

After:
  I0207 19:38:32.505213 32039 debug-util-test.cc:175] Throughput: 57916 dumps/second (symbolize=0)
  I0207 19:38:33.506417 32039 debug-util-test.cc:175] Throughput: 595 dumps/second (symbolize=1)

Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
---
M src/kudu/util/debug-util.cc
1 file changed, 70 insertions(+), 16 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/9254/6
-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 6
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Mike Percy, Kudu Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/9254

to look at the new patch set (#2).

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................

KUDU-2291 (part 3): use futex to speed up stack collection

Previously the thread which requests a stack trace would poll on an
atomic variable waiting for the stack to be collected. Of course
spinning is too CPU-hungry so it would actually sleep for 10ms in
each iteration. This limited the performance of stack collection to
about 100/second, which is problematic for the /stacks web page on a
server that may have thousands of threads.

This switches to using the futex system call to allow the dumper thread
to sleep and the dumpee thread to wake it back up when the results are
ready.

Why do we use the low level futex and not something like a condition
variable or CountdownLatch? The issue is that the dumpee thread is
running in a signal handler context, and most things are not safe to do
in that context. Particularly, mutexes may block, which is very naughty,
and condition variables and latches are currently implemented on top of
mutexes. pthread semaphores are another option, but those don't support
proper timeout semantics, which is a feature we use in this use case.

Whether the futex syscall itself is async-signal-safe is up for some debate.
Clearly the "wait" mode is not, since it may block, but is the "wake" mode
that we use here allowed? Technically I don't think so. In practice,
the pthread sem_post() API _is_ documented to be async-signal-safe and
I verified that in libc source it's implemented using futex to wake the
waiter. So, if we are doing something illegal, at least we're doing
the same illegal thing as libc, so we're very unlikely to ever break.

Obviously futex is not portable to macOS, but this code was already
Linux-specific so we haven't lost much.

This speeds up the benchmark for stack trace collection (without symbolization)
by 585x:

Before:
  I0207 19:33:20.960477 30799 debug-util-test.cc:175] Throughput: 99 dumps/second (symbolize=0)
  I0207 19:33:21.963946 30799 debug-util-test.cc:175] Throughput: 84 dumps/second (symbolize=1)

After:
  I0207 19:38:32.505213 32039 debug-util-test.cc:175] Throughput: 57916 dumps/second (symbolize=0)
  I0207 19:38:33.506417 32039 debug-util-test.cc:175] Throughput: 595 dumps/second (symbolize=1)

Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
---
M src/kudu/util/debug-util.cc
1 file changed, 68 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/9254/2
-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 2
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has removed Kudu Jenkins from this change.  ( http://gerrit.cloudera.org:8080/9254 )

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................


Removed reviewer Kudu Jenkins with the following votes:

* Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteReviewer
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/9254 )

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/9254/5/src/kudu/util/debug-util.cc
File src/kudu/util/debug-util.cc:

http://gerrit.cloudera.org:8080/#/c/9254/5/src/kudu/util/debug-util.cc@144
PS5, Line 144: 0
how about: nullptr /* ignored */


http://gerrit.cloudera.org:8080/#/c/9254/5/src/kudu/util/debug-util.cc@197
PS5, Line 197: spins waiting
s/spins waiting/waits/



-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 21 Feb 2018 00:37:42 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Will Berkeley (Code Review)" <ge...@cloudera.org>.
Will Berkeley has posted comments on this change. ( http://gerrit.cloudera.org:8080/9254 )

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................


Patch Set 3: Code-Review+1

(1 comment)

LGTM with one comment nit, but I'd like Mike to take a look since futex semantics are new to me.

http://gerrit.cloudera.org:8080/#/c/9254/3/src/kudu/util/debug-util.cc
File src/kudu/util/debug-util.cc:

http://gerrit.cloudera.org:8080/#/c/9254/3/src/kudu/util/debug-util.cc@331
PS3, Line 331: a few iterations of the loop
nit: The loop is gone.



-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 13 Feb 2018 20:55:10 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Mike Percy (Code Review)" <ge...@cloudera.org>.
Mike Percy has posted comments on this change. ( http://gerrit.cloudera.org:8080/9254 )

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................


Patch Set 6: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 6
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 21 Feb 2018 03:07:21 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Hello Will Berkeley, Mike Percy, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/9254

to look at the new patch set (#4).

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................

KUDU-2291 (part 3): use futex to speed up stack collection

Previously the thread which requests a stack trace would poll on an
atomic variable waiting for the stack to be collected. Of course
spinning is too CPU-hungry so it would actually sleep for 10ms in
each iteration. This limited the performance of stack collection to
about 100/second, which is problematic for the /stacks web page on a
server that may have thousands of threads.

This switches to using the futex system call to allow the dumper thread
to sleep and the dumpee thread to wake it back up when the results are
ready.

Why do we use the low level futex and not something like a condition
variable or CountdownLatch? The issue is that the dumpee thread is
running in a signal handler context, and most things are not safe to do
in that context. Particularly, mutexes may block, which is very naughty,
and condition variables and latches are currently implemented on top of
mutexes. pthread semaphores are another option, but those don't support
proper timeout semantics, which is a feature we use in this use case.

Whether the futex syscall itself is async-signal-safe is up for some debate.
Clearly the "wait" mode is not, since it may block, but is the "wake" mode
that we use here allowed? Technically I don't think so. In practice,
the pthread sem_post() API _is_ documented to be async-signal-safe and
I verified that in libc source it's implemented using futex to wake the
waiter. So, if we are doing something illegal, at least we're doing
the same illegal thing as libc, so we're very unlikely to ever break.

Obviously futex is not portable to macOS, but this code was already
Linux-specific so we haven't lost much.

This speeds up the benchmark for stack trace collection (without symbolization)
by 585x:

Before:
  I0207 19:33:20.960477 30799 debug-util-test.cc:175] Throughput: 99 dumps/second (symbolize=0)
  I0207 19:33:21.963946 30799 debug-util-test.cc:175] Throughput: 84 dumps/second (symbolize=1)

After:
  I0207 19:38:32.505213 32039 debug-util-test.cc:175] Throughput: 57916 dumps/second (symbolize=0)
  I0207 19:38:33.506417 32039 debug-util-test.cc:175] Throughput: 595 dumps/second (symbolize=1)

Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
---
M src/kudu/util/debug-util.cc
1 file changed, 69 insertions(+), 15 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/54/9254/4
-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 4
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/9254 )

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/9254/5/src/kudu/util/debug-util.cc
File src/kudu/util/debug-util.cc:

http://gerrit.cloudera.org:8080/#/c/9254/5/src/kudu/util/debug-util.cc@144
PS5, Line 144: 0
> how about: nullptr /* ignored */
Done


http://gerrit.cloudera.org:8080/#/c/9254/5/src/kudu/util/debug-util.cc@197
PS5, Line 197: spins waiting
> s/spins waiting/waits/
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 5
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Wed, 21 Feb 2018 00:49:59 +0000
Gerrit-HasComments: Yes

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/9254 )

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................


Patch Set 3: Verified+1

Jenkins issue was a flake which I fixed separately (UBSAN issue)


-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>
Gerrit-Comment-Date: Tue, 13 Feb 2018 20:35:02 +0000
Gerrit-HasComments: No

[kudu-CR] KUDU-2291 (part 3): use futex to speed up stack collection

Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/9254 )

Change subject: KUDU-2291 (part 3): use futex to speed up stack collection
......................................................................

KUDU-2291 (part 3): use futex to speed up stack collection

Previously the thread which requests a stack trace would poll on an
atomic variable waiting for the stack to be collected. Of course
spinning is too CPU-hungry so it would actually sleep for 10ms in
each iteration. This limited the performance of stack collection to
about 100/second, which is problematic for the /stacks web page on a
server that may have thousands of threads.

This switches to using the futex system call to allow the dumper thread
to sleep and the dumpee thread to wake it back up when the results are
ready.

Why do we use the low level futex and not something like a condition
variable or CountdownLatch? The issue is that the dumpee thread is
running in a signal handler context, and most things are not safe to do
in that context. Particularly, mutexes may block, which is very naughty,
and condition variables and latches are currently implemented on top of
mutexes. pthread semaphores are another option, but those don't support
proper timeout semantics, which is a feature we use in this use case.

Whether the futex syscall itself is async-signal-safe is up for some debate.
Clearly the "wait" mode is not, since it may block, but is the "wake" mode
that we use here allowed? Technically I don't think so. In practice,
the pthread sem_post() API _is_ documented to be async-signal-safe and
I verified that in libc source it's implemented using futex to wake the
waiter. So, if we are doing something illegal, at least we're doing
the same illegal thing as libc, so we're very unlikely to ever break.

Obviously futex is not portable to macOS, but this code was already
Linux-specific so we haven't lost much.

This speeds up the benchmark for stack trace collection (without symbolization)
by 585x:

Before:
  I0207 19:33:20.960477 30799 debug-util-test.cc:175] Throughput: 99 dumps/second (symbolize=0)
  I0207 19:33:21.963946 30799 debug-util-test.cc:175] Throughput: 84 dumps/second (symbolize=1)

After:
  I0207 19:38:32.505213 32039 debug-util-test.cc:175] Throughput: 57916 dumps/second (symbolize=0)
  I0207 19:38:33.506417 32039 debug-util-test.cc:175] Throughput: 595 dumps/second (symbolize=1)

Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Reviewed-on: http://gerrit.cloudera.org:8080/9254
Reviewed-by: Mike Percy <mp...@apache.org>
Tested-by: Kudu Jenkins
---
M src/kudu/util/debug-util.cc
1 file changed, 70 insertions(+), 16 deletions(-)

Approvals:
  Mike Percy: Looks good to me, approved
  Kudu Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/9254
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icb772f0607275ec340c7f38be05347fc14820cbd
Gerrit-Change-Number: 9254
Gerrit-PatchSet: 7
Gerrit-Owner: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Mike Percy <mp...@apache.org>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Reviewer: Will Berkeley <wd...@gmail.com>