You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2020/02/28 05:00:18 UTC

[kudu-CR] [java] subprocess: add a metrics message

Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/15315


Change subject: [java] subprocess: add a metrics message
......................................................................

[java] subprocess: add a metrics message

This patch adds a metrics message to each SubprocessResponsePB. These
metrics are general metrics that most Subprocess implementations will
likely find useful. Namely, inbound queue time, outbound queue time,
inbound queue length, outbound queue length, and execution time.

To get these metrics, I've passed the new SubprocessMetrics class around
alongside the request/response as it journeys through the
SubprocessExecutor, timing the different stages as it goes. Right before
sending the final response over the pipe, the queue lengths are
determined and the final response is built.

A couple of tests are added to demonstrate how these metrics may be
useful. Along the way, I refactored the EchoSubprocess tests to be a bit
more ergonomic w.r.t creating, sending, and receiving messages.

These metrics are currently unused -- a later patch will plumb them into
the C++ metrics.

Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
---
M java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/OutboundResponse.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
M java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
M src/kudu/subprocess/subprocess.proto
11 files changed, 558 insertions(+), 145 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/15315/1
-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:

http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto@49
PS1, Line 49:   // Number of requests in the subprocess's inbound queue.
            :   optional int64 inbound_queue_length = 3;
            : 
            :   // Number of responses in the subprocess's outbound queue.
            :   optional int64 outbound_queue_length = 4;
> To get a gauge what the current state of the queues are in general, "as late as we can before sending back to C++" is useful.

Not following the intuition here: doesn't _any_ point in time yield "the current state of the queues", as long as the point in time is consistently applied (i.e. always measure at enqueue time)?

> Do you think it's worth adding both? A snapshot upon entering the system and a snapshot upon exiting the system? Or is that overkill and wasteful?

I think that's overkill, at least for now. We can always revisit it in the future.



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 28 Feb 2020 22:52:58 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 2: Code-Review+1

(2 comments)

Overall looks good to me, just a few nits. Thanks a lot Andrew for working on the patch!

http://gerrit.cloudera.org:8080/#/c/15315/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java:

http://gerrit.cloudera.org:8080/#/c/15315/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java@77
PS2, Line 77: reset
So this seems there is an order assumption about calling recordInboundQueueTimeMs, recordExecutionTimeMs and recordOutboundQueueTimeMs. Do you think that needs to be documented?


http://gerrit.cloudera.org:8080/#/c/15315/2/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:

http://gerrit.cloudera.org:8080/#/c/15315/2/src/kudu/subprocess/subprocess.proto@54
PS2, Line 54: .
nit: remove.



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 03 Mar 2020 01:25:57 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 1:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java@69
PS1, Line 69:       metrics.startTimer();
Note in a comment here what we're starting to record and who is expected to finish the recording.


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java@97
PS1, Line 97:   private SubprocessResponsePB.Builder getResponseBuilder(byte[] data) {
It's not obvious from the name that this also executes the request. Could you rename it (and update the Javadoc too)?


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@83
PS1, Line 83:       metrics.startTimer();
Note in a comment here what we're starting to record and who is expected to finish the recording.


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java@64
PS1, Line 64:   protected abstract ResponseT createResponse(RequestT request);
This too involves executing the request; could you rename and update the Javadoc to make that more explicit?


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:

PS1: 
One thing I find a little unintuitive with the current implementation is the asymmetry of the two queues: the inbound queue contains encoded request data, while the outbound queue contains decoded response data.

Can we address that, either by parsing the requests when we read them off the wire, or encoding the responses after execution? On further thought, the latter won't work if there are additional metrics to calculate at the time that we write to stdout.


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@95
PS1, Line 95:     BlockingQueue<InboundRequest> inboundQueue =
            :         new ArrayBlockingQueue<>(queueSize, /* fair= */true);
For symmetry, how do you feel about declaring this as a class field alongside outboundQueue?


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java@57
PS1, Line 57:   public void recordInboundQueueTimeMs() {
In the various record functions could we precondition on stopwatch.isRunning()?


http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:

http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto@49
PS1, Line 49:   // Number of requests in the subprocess's inbound queue.
            :   optional int64 inbound_queue_length = 3;
            : 
            :   // Number of responses in the subprocess's outbound queue.
            :   optional int64 outbound_queue_length = 4;
"...at the time that the {request,response} was queued." presumably?

Having read through the Java code, this will actually reflect the queue lengths at the time that the response is sent back. Which is more useful? "At the time of queueing" seems more intuitive to me, and it's what we do in src/kudu/util/threadpool.{cc,h}.



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 28 Feb 2020 19:47:07 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 1:

(9 comments)

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java@21
PS1, Line 21: It is expected that the
            :  * <code>SubprocessMetrics</code> have begun timing the time it takes to make
            :  * it through the queue
> No, it's only required that the timer is started before adding it to the qu
To make it more obvious, I've commented that it's required, even if it's not a strict requirement.


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java@69
PS1, Line 69:       metrics.startTimer();
> +1, as of now it is a bit confusing that we call startTimer twice here.
Done


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java@97
PS1, Line 97:   private SubprocessResponsePB.Builder getResponseBuilder(byte[] data) {
> It's not obvious from the name that this also executes the request. Could y
Done


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java@83
PS1, Line 83:       metrics.startTimer();
> Note in a comment here what we're starting to record and who is expected to
Done


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java@64
PS1, Line 64:   protected abstract ResponseT createResponse(RequestT request);
> This too involves executing the request; could you rename and update the Ja
Done


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:

PS1: 
> Yeah, I think the guiding principle here was "do as little as we can in the
I'm going to punt on this unless you feel strongly about it. I'm pretty sure that in parsing, we're doing a decent amount of work, so I'd rather keep it in the parser threads.

And you're right about the metrics calculation -- I don't see a great way to write the metrics after already serializing the response to a byte array.


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java@95
PS1, Line 95:     BlockingQueue<InboundRequest> inboundQueue =
            :         new ArrayBlockingQueue<>(queueSize, /* fair= */true);
> For symmetry, how do you feel about declaring this as a class field alongsi
Done


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java@57
PS1, Line 57:   public void recordInboundQueueTimeMs() {
> In the various record functions could we precondition on stopwatch.isRunnin
Done


http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:

http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto@49
PS1, Line 49:   // Number of requests in the subprocess's inbound queue.
            :   optional int64 inbound_queue_length = 3;
            : 
            :   // Number of responses in the subprocess's outbound queue.
            :   optional int64 outbound_queue_length = 4;
> Oh, I think I see what you mean. And yeah, the threadpool metric is a histo
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 02 Mar 2020 23:28:23 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:

PS1: 
> One thing I find a little unintuitive with the current implementation is th
Yeah, I think the guiding principle here was "do as little as we can in the reader/writer threads since they're single-threaded". I'll need to look into whether we can maybe construct a SubprocessRequestPB.Builder without entirely deserializing the bytes on the reader thread.


http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:

http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto@49
PS1, Line 49:   // Number of requests in the subprocess's inbound queue.
            :   optional int64 inbound_queue_length = 3;
            : 
            :   // Number of responses in the subprocess's outbound queue.
            :   optional int64 outbound_queue_length = 4;
> "...at the time that the {request,response} was queued." presumably?
To debug slowness of a single request, a snapshot at the time of queueing is useful. I don't expect us to actually debug a single request in isolation though.

To get a gauge what the current state of the queues are in general, "as late as we can before sending back to C++" is useful.

Do you think it's worth adding both? A snapshot upon entering the system and a snapshot upon exiting the system? Or is that overkill and wasteful?



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Fri, 28 Feb 2020 19:56:25 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:

http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto@49
PS1, Line 49:   // Number of requests in the subprocess's inbound queue.
            :   optional int64 inbound_queue_length = 3;
            : 
            :   // Number of responses in the subprocess's outbound queue.
            :   optional int64 outbound_queue_length = 4;
> > To get a gauge what the current state of the queues are in general, "as l
>Not following the intuition here: doesn't _any_ point in time yield "the current state of the queues", as long as the point in time is consistently applied (i.e. always measure at enqueue time)?

It being consistent make the metrics usable, but take this as an example:
- enqueue on inbound, req 1 (take snapshot of queues at t=1000)
- enqueue on inbound, req 2 (take snapshot of queues at t=1001)
- enqueue on inbound, req 3 (take snapshot of queues at t=1002)
- process the requests in parallel
- enqueue on outbound, resp 3
- enqueue on outbound, resp 1
- enqueue on outbound, resp 2

This sequence gives us an unstable view of the state of the queue if we want to use this as a Gauge metric that tracks the current length of the queue. Though if we want to expose this as a histogram, the ordering doesn't matter as much.



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 29 Feb 2020 00:20:54 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has removed a vote on this change.

Change subject: [java] subprocess: add a metrics message
......................................................................


Removed Verified-1 by Kudu Jenkins (120)
-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: deleteVote
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................

[java] subprocess: add a metrics message

This patch adds a metrics message to each SubprocessResponsePB. These
metrics are general metrics that most Subprocess implementations will
likely find useful. Namely, inbound queue time, outbound queue time,
inbound queue length, outbound queue length, and execution time.

To get these metrics, I've passed the new SubprocessMetrics class around
alongside the request/response as it journeys through the
SubprocessExecutor, timing the different stages as it goes. Right before
sending the final response over the pipe, the queue lengths are
determined and the final response is built.

A couple of tests are added to demonstrate how these metrics may be
useful. Along the way, I refactored the EchoSubprocess tests to be a bit
more ergonomic w.r.t creating, sending, and receiving messages.

Along the way, I renamed some of the subprocess methods for clarity:
- ProtocolHandler.handleRequest() -> unpackAndExecuteRequest()
- ProtocolHandler.createResponse() -> executeRequest()
- MessageParser.getResponse() -> parseAndExecuteRequest()

These metrics are currently unused -- a later patch will plumb them into
the C++ metrics.

Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Reviewed-on: http://gerrit.cloudera.org:8080/15315
Reviewed-by: Adar Dembo <ad...@cloudera.com>
Tested-by: Andrew Wong <aw...@cloudera.com>
---
M java/kudu-subprocess-echo/src/main/java/org/apache/kudu/subprocess/echo/EchoProtocolHandler.java
M java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/OutboundResponse.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
M java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
M src/kudu/subprocess/subprocess.proto
12 files changed, 593 insertions(+), 162 deletions(-)

Approvals:
  Adar Dembo: Looks good to me, approved
  Andrew Wong: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:

PS1: 
> Maybe a TODO then?
Done in InboundRequest.java



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 03 Mar 2020 05:28:29 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto
File src/kudu/subprocess/subprocess.proto:

http://gerrit.cloudera.org:8080/#/c/15315/1/src/kudu/subprocess/subprocess.proto@49
PS1, Line 49:   // Number of requests in the subprocess's inbound queue.
            :   optional int64 inbound_queue_length = 3;
            : 
            :   // Number of responses in the subprocess's outbound queue.
            :   optional int64 outbound_queue_length = 4;
> >Not following the intuition here: doesn't _any_ point in time yield "the c
Oh, I think I see what you mean. And yeah, the threadpool metric is a histogram.



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Sat, 29 Feb 2020 00:31:33 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 1:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java
File java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java@122
PS1, Line 122: Sends
> nit: Send.
Not sure I agree here, given we describe methods with adjectives instead of verbs elsewhere. Or is there something about this in a style guide that I missed?


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java@128
PS1, Line 128: Receives
> nit: Receive
Not sure I agree here, given we describe methods with adjectives instead of verbs elsewhere. Or is there something about this in a style guide that I missed?


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java@135
PS1, Line 135: Sets
> nit: Set
Not sure I agree here, given we describe methods with adjectives instead of verbs elsewhere. Or is there something about this in a style guide that I missed?


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java@21
PS1, Line 21: It is expected that the
            :  * <code>SubprocessMetrics</code> have begun timing the time it takes to make
            :  * it through the queue
> Is this required SubprocessMetrics has started timing before the initializa
No, it's only required that the timer is started before adding it to the queue.


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java@71
PS1, Line 71: reset
> Why use reset not stop here?
I'll update the comment, but reset() lets us begin timing a new section the next time we start the timer. If we call stop(), the timer will carry on from where it left off (at least I assume so, based on the docs), which isn't what we want if we're timing non-overlapping periods of time.

https://guava.dev/releases/19.0/api/docs/com/google/common/base/Stopwatch.html



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 02 Mar 2020 07:58:48 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 03 Mar 2020 05:30:32 +0000
Gerrit-HasComments: No

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Adar Dembo, Hao Hao, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/15315

to look at the new patch set (#2).

Change subject: [java] subprocess: add a metrics message
......................................................................

[java] subprocess: add a metrics message

This patch adds a metrics message to each SubprocessResponsePB. These
metrics are general metrics that most Subprocess implementations will
likely find useful. Namely, inbound queue time, outbound queue time,
inbound queue length, outbound queue length, and execution time.

To get these metrics, I've passed the new SubprocessMetrics class around
alongside the request/response as it journeys through the
SubprocessExecutor, timing the different stages as it goes. Right before
sending the final response over the pipe, the queue lengths are
determined and the final response is built.

A couple of tests are added to demonstrate how these metrics may be
useful. Along the way, I refactored the EchoSubprocess tests to be a bit
more ergonomic w.r.t creating, sending, and receiving messages.

Along the way, I renamed some of the subprocess methods for clarity:
- ProtocolHandler.handleRequest() -> unpackAndExecuteRequest()
- ProtocolHandler.createResponse() -> executeRequest()
- MessageParser.getResponse() -> parseAndExecuteRequest()

These metrics are currently unused -- a later patch will plumb them into
the C++ metrics.

Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
---
M java/kudu-subprocess-echo/src/main/java/org/apache/kudu/subprocess/echo/EchoProtocolHandler.java
M java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/OutboundResponse.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
M java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
M src/kudu/subprocess/subprocess.proto
12 files changed, 591 insertions(+), 162 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/15315/2
-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Hao Hao (Code Review)" <ge...@cloudera.org>.
Hao Hao has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 1:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java
File java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java@122
PS1, Line 122: Sends
nit: Send.


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java@128
PS1, Line 128: Receives
nit: Receive


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java@135
PS1, Line 135: Sets
nit: Set


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java@21
PS1, Line 21: It is expected that the
            :  * <code>SubprocessMetrics</code> have begun timing the time it takes to make
            :  * it through the queue
Is this required SubprocessMetrics has started timing before the initialization of InboundRequest?


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java@69
PS1, Line 69:       metrics.startTimer();
> Note in a comment here what we're starting to record and who is expected to
+1, as of now it is a bit confusing that we call startTimer twice here.


http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java:

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java@71
PS1, Line 71: reset
Why use reset not stop here?



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Mon, 02 Mar 2020 07:41:22 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/15315/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java:

http://gerrit.cloudera.org:8080/#/c/15315/2/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java@77
PS2, Line 77: reset
> So this seems there is an order assumption about calling recordInboundQueue
Not really. I don't think it'd add anything to what's already documented.



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 03 Mar 2020 01:30:55 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 3: Verified+1

Unrelated flake of disk_failure-itest


-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 03 Mar 2020 06:19:13 +0000
Gerrit-HasComments: No

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Adar Dembo (Code Review)" <ge...@cloudera.org>.
Adar Dembo has posted comments on this change. ( http://gerrit.cloudera.org:8080/15315 )

Change subject: [java] subprocess: add a metrics message
......................................................................


Patch Set 2: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/15315/1/java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
File java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java:

PS1: 
> I'm going to punt on this unless you feel strongly about it. I'm pretty sur
Maybe a TODO then?



-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Comment-Date: Tue, 03 Mar 2020 04:17:21 +0000
Gerrit-HasComments: Yes

[kudu-CR] [java] subprocess: add a metrics message

Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello Attila Bukor, Kudu Jenkins, Adar Dembo, Hao Hao, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/15315

to look at the new patch set (#3).

Change subject: [java] subprocess: add a metrics message
......................................................................

[java] subprocess: add a metrics message

This patch adds a metrics message to each SubprocessResponsePB. These
metrics are general metrics that most Subprocess implementations will
likely find useful. Namely, inbound queue time, outbound queue time,
inbound queue length, outbound queue length, and execution time.

To get these metrics, I've passed the new SubprocessMetrics class around
alongside the request/response as it journeys through the
SubprocessExecutor, timing the different stages as it goes. Right before
sending the final response over the pipe, the queue lengths are
determined and the final response is built.

A couple of tests are added to demonstrate how these metrics may be
useful. Along the way, I refactored the EchoSubprocess tests to be a bit
more ergonomic w.r.t creating, sending, and receiving messages.

Along the way, I renamed some of the subprocess methods for clarity:
- ProtocolHandler.handleRequest() -> unpackAndExecuteRequest()
- ProtocolHandler.createResponse() -> executeRequest()
- MessageParser.getResponse() -> parseAndExecuteRequest()

These metrics are currently unused -- a later patch will plumb them into
the C++ metrics.

Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
---
M java/kudu-subprocess-echo/src/main/java/org/apache/kudu/subprocess/echo/EchoProtocolHandler.java
M java/kudu-subprocess-echo/src/test/java/org/apache/kudu/subprocess/echo/TestEchoSubprocess.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/InboundRequest.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageParser.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageReader.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/MessageWriter.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/OutboundResponse.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/ProtocolHandler.java
M java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessExecutor.java
A java/kudu-subprocess/src/main/java/org/apache/kudu/subprocess/SubprocessMetrics.java
M java/kudu-subprocess/src/test/java/org/apache/kudu/subprocess/MessageTestUtil.java
M src/kudu/subprocess/subprocess.proto
12 files changed, 593 insertions(+), 162 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/15/15315/3
-- 
To view, visit http://gerrit.cloudera.org:8080/15315
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I11a89fff8df23c5057c577f2aebfd40922d01e3c
Gerrit-Change-Number: 15315
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Adar Dembo <ad...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Attila Bukor <ab...@apache.org>
Gerrit-Reviewer: Hao Hao <ha...@cloudera.com>
Gerrit-Reviewer: Kudu Jenkins (120)