You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Andrew Wong (Code Review)" <ge...@cloudera.org> on 2018/06/12 04:09:55 UTC
[kudu-CR] parse metrics log: update to the new format
Andrew Wong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10693
Change subject: parse_metrics_log: update to the new format
......................................................................
parse_metrics_log: update to the new format
The new diagnostics logs report more than just metrics, and thus, output
a bit differently than they did in simpler times. This patch updates the
parsing to be compatible with the new logs.
Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
---
M src/kudu/scripts/parse_metrics_log.py
1 file changed, 7 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/10693/1
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 2: Code-Review-1
Agh, may have just misunderstood the histogram bits.
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Comment-Date: Tue, 12 Jun 2018 23:46:56 +0000
Gerrit-HasComments: No
[kudu-CR] parse metrics log: update to the new format
Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Comment-Date: Tue, 12 Jun 2018 14:39:02 +0000
Gerrit-HasComments: No
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 2:
> Patch Set 2:
>
> (4 comments)
>
> I guess the histogram-parsing code was "off" because now we only include buckets that have changed/are non-zero?
>
> Did you test it against some actual logs to make sure it has the expected results?
>
> Also, wondering if we should bite the bullet soon and port this to C++ so it runs faster and ships with kudu.
Re: the histogram-parsing code, I'm confused about why it was calculating the percentiles using the delta between the previous and current snapshots. I would've expected it to be the aggregate of the two instead.
E.g. in the case the total count stayed the same between snapshots, but the distribution changed, we would return UNKNOWN_PERCENTILES, but it seems reasonable still compute percentiles since we still have some distribution of the metric at that time, no?
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 13 Jun 2018 00:44:10 +0000
Gerrit-HasComments: No
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello David Ribeiro Alves, Kudu Jenkins, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10693
to look at the new patch set (#3).
Change subject: parse_metrics_log: update to the new format
......................................................................
parse_metrics_log: update to the new format
The new diagnostics logs report more than just metrics, and thus, output
a bit differently than they did in simpler times. This patch updates the
parsing to be compatible with the new logs.
The changes this patch addresses:
- each metrics log line had a different number of fields
- the metrics log will only emit metrics from entities that have
changed; as such, the processing of metrics has been changed; where we
previously filled in NaNs for missing data, we now pull from the
previous snapshot, or fill in 0 if one doesn't exist
Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
---
M src/kudu/scripts/parse_metrics_log.py
1 file changed, 125 insertions(+), 62 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/10693/3
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] parse metrics log: update to the new format
Posted by "David Ribeiro Alves (Code Review)" <ge...@cloudera.org>.
David Ribeiro Alves has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 25 Jun 2018 18:00:03 +0000
Gerrit-HasComments: No
[kudu-CR] parse metrics log: update to the new format
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 4: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 25 Jun 2018 22:40:40 +0000
Gerrit-HasComments: No
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello David Ribeiro Alves, Kudu Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10693
to look at the new patch set (#2).
Change subject: parse_metrics_log: update to the new format
......................................................................
parse_metrics_log: update to the new format
The new diagnostics logs report more than just metrics, and thus, output
a bit differently than they did in simpler times. This patch updates the
parsing to be compatible with the new logs.
The changes this patch addresses:
- each metrics log line had a different number of fields
- the metrics log will only emit metrics that have changed; as such,
the processing of metrics has been changed; where we previously filled
in NaNs for missing data, we now pull from the previous snapshot, or
fill in 0 if one doesn't exist
- the histogram-parsing code seemed a bit off; this patch updates it
Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
---
M src/kudu/scripts/parse_metrics_log.py
1 file changed, 64 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/10693/2
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 1:
There's a bit more to this change that needs updating; namely handling this change:
https://github.com/apache/kudu/commit/ab5387403e42df3161492d6f48b9b51bc85c7499#diff-dc9991e75fccdaad07446db734ad6712
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 1
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Comment-Date: Tue, 12 Jun 2018 22:03:08 +0000
Gerrit-HasComments: No
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 3:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py
File src/kudu/scripts/parse_metrics_log.py:
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py@146
PS2, Line 146: # Iterate through all the entities for this metric.
> seems somewhat unexpected for this function to have a side effect on its ar
Done
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py@147
PS2, Line 147: for eid in eid_to_vals:
> "m not in cur"
Done
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py@163
PS2, Line 163: if metric_name in ret:
> same
Done
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py@169
PS2, Line 169: def histogram_stats(aggregated_prev, aggregated_cur, m):
> same comment about unexpected side effects
Done
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Thu, 21 Jun 2018 21:23:42 +0000
Gerrit-HasComments: Yes
[kudu-CR] parse metrics log: update to the new format
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10693/3/src/kudu/scripts/parse_metrics_log.py
File src/kudu/scripts/parse_metrics_log.py:
http://gerrit.cloudera.org:8080/#/c/10693/3/src/kudu/scripts/parse_metrics_log.py@123
PS3, Line 123: # Add the metric_id to the metrics map.
you could use:
ret = collections.defaultdict(dict)
and it'll magically create empty dictionaries upon first reference, avoiding this if statement
http://gerrit.cloudera.org:8080/#/c/10693/3/src/kudu/scripts/parse_metrics_log.py@276
PS3, Line 276: if not log_type == "metrics":
nit: log_type != "metrics"
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 3
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 25 Jun 2018 22:16:40 +0000
Gerrit-HasComments: Yes
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Hello David Ribeiro Alves, Kudu Jenkins, Todd Lipcon,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/10693
to look at the new patch set (#4).
Change subject: parse_metrics_log: update to the new format
......................................................................
parse_metrics_log: update to the new format
The new diagnostics logs report more than just metrics, and thus, output
a bit differently than they did in simpler times. This patch updates the
parsing to be compatible with the new logs.
The changes this patch addresses:
- each metrics log line had a different number of fields
- the metrics log will only emit metrics from entities that have
changed; as such, the processing of metrics has been changed; where we
previously filled in NaNs for missing data, we now pull from the
previous snapshot, or fill in 0 if one doesn't exist
I tested this manually by running against some metrics logs I collected
across a couple of workloads. The results seem to match what was
expected.
Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
---
M src/kudu/scripts/parse_metrics_log.py
1 file changed, 124 insertions(+), 67 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/93/10693/4
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10693/3/src/kudu/scripts/parse_metrics_log.py
File src/kudu/scripts/parse_metrics_log.py:
http://gerrit.cloudera.org:8080/#/c/10693/3/src/kudu/scripts/parse_metrics_log.py@123
PS3, Line 123:
> you could use:
Neat
http://gerrit.cloudera.org:8080/#/c/10693/3/src/kudu/scripts/parse_metrics_log.py@276
PS3, Line 276: raise Exception("timestamps must be in ascending order (%f <= %f at %s:%d)"
> nit: log_type != "metrics"
Done
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 4
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Mon, 25 Jun 2018 22:32:37 +0000
Gerrit-HasComments: Yes
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 2:
Ah, this may explain it (from the docs):
NOTE: All histograms and counters are measured since the server start time, and are not reset upon collection.
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 13 Jun 2018 00:57:23 +0000
Gerrit-HasComments: No
[kudu-CR] parse metrics log: update to the new format
Posted by "Todd Lipcon (Code Review)" <ge...@cloudera.org>.
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 2:
(4 comments)
I guess the histogram-parsing code was "off" because now we only include buckets that have changed/are non-zero?
Did you test it against some actual logs to make sure it has the expected results?
Also, wondering if we should bite the bullet soon and port this to C++ so it runs faster and ships with kudu.
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py
File src/kudu/scripts/parse_metrics_log.py:
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py@146
PS2, Line 146: # Update the current metric in case we have missing entries.
seems somewhat unexpected for this function to have a side effect on its arguments. Is there another spot we can put this, eg dpwn where we do the actual reading of the lines, we could shallow-copy the previous data and merge in the new data?
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py@147
PS2, Line 147: if not m in cur:
"m not in cur"
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py@163
PS2, Line 163: if not m in prev:
same
http://gerrit.cloudera.org:8080/#/c/10693/2/src/kudu/scripts/parse_metrics_log.py@169
PS2, Line 169: cur[m] = prev[m]
same comment about unexpected side effects
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>
Gerrit-Comment-Date: Wed, 13 Jun 2018 00:11:26 +0000
Gerrit-HasComments: Yes
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
Patch Set 2: -Code-Review
Actually nevermind, pretty sure the update to histogram_stats() makes sense, open to feedback on it.
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 2
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Comment-Date: Tue, 12 Jun 2018 23:55:06 +0000
Gerrit-HasComments: No
[kudu-CR] parse metrics log: update to the new format
Posted by "Andrew Wong (Code Review)" <ge...@cloudera.org>.
Andrew Wong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10693 )
Change subject: parse_metrics_log: update to the new format
......................................................................
parse_metrics_log: update to the new format
The new diagnostics logs report more than just metrics, and thus, output
a bit differently than they did in simpler times. This patch updates the
parsing to be compatible with the new logs.
The changes this patch addresses:
- each metrics log line had a different number of fields
- the metrics log will only emit metrics from entities that have
changed; as such, the processing of metrics has been changed; where we
previously filled in NaNs for missing data, we now pull from the
previous snapshot, or fill in 0 if one doesn't exist
I tested this manually by running against some metrics logs I collected
across a couple of workloads. The results seem to match what was
expected.
Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Reviewed-on: http://gerrit.cloudera.org:8080/10693
Reviewed-by: Todd Lipcon <to...@apache.org>
Tested-by: Kudu Jenkins
---
M src/kudu/scripts/parse_metrics_log.py
1 file changed, 124 insertions(+), 67 deletions(-)
Approvals:
Todd Lipcon: Looks good to me, approved
Kudu Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/10693
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: If11b7ecc93a3f64db3b7c1994f47308b3ec44029
Gerrit-Change-Number: 10693
Gerrit-PatchSet: 5
Gerrit-Owner: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: Andrew Wong <aw...@cloudera.com>
Gerrit-Reviewer: David Ribeiro Alves <da...@gmail.com>
Gerrit-Reviewer: Kudu Jenkins
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>