You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kudu.apache.org by to...@apache.org on 2016/01/12 02:50:05 UTC
incubator-kudu git commit: parse_test_failure: optimization and usage
improvement
Repository: incubator-kudu
Updated Branches:
refs/heads/master ad9943376 -> 556bf3b8d
parse_test_failure: optimization and usage improvement
This adds a couple micro-optimizations to the parse_test_failure script
which substantially increase performance. I noticed that, now that we are
running the test result server on the public internet, Google found it
and is crawling the 'diagnose' pages for failed tests. Currently, those
pages can take 10s of seconds (or worse) to load, which ends up spinning
the CPU and blocking other legit usage of the server, since it's running
single-threaded at the moment. This likely has affected other test runs
and prevented them from fetching the flaky-test-list or reporting flakes.
The micro-optimizations are simple: just using substring matches to do quick
checks before running the more expensive regular expressions. I tested by
running the script on a recent client-test.txt log I had lying around with
about 28000 lines.
I tested this in production like a good web developer. It's deployed and
seems to be working (plus verified that the 'diagnose' links load quickly
now).
before:
Performance counter stats for './build-support/parse_test_failure.py ./build/test-logs/client-test.txt':
11732.165505 task-clock (msec) # 1.001 CPUs utilized
119 context-switches # 0.010 K/sec
1 cpu-migrations # 0.000 K/sec
1,805 page-faults # 0.154 K/sec
38,840,157,152 cycles # 3.311 GHz
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
103,675,773,467 instructions # 2.67 insns per cycle
20,562,530,240 branches # 1752.663 M/sec
45,622,875 branch-misses # 0.22% of all branches
11.724291727 seconds time elapsed
after:
Performance counter stats for './build-support/parse_test_failure.py ./build/test-logs/client-test.txt' (10 runs):
73.823229 task-clock (msec) # 1.009 CPUs utilized
1 context-switches # 0.014 K/sec
1 cpu-migrations # 0.014 K/sec
1,901 page-faults # 0.026 M/sec
243,137,251 cycles # 3.332 GHz
<not supported> stalled-cycles-frontend
<not supported> stalled-cycles-backend
513,003,074 instructions # 2.14 insns per cycle
128,224,780 branches # 1757.295 M/sec
1,149,019 branch-misses # 0.90% of all branches
0.073158565 seconds time elapsed ( +- 0.50% )
Change-Id: Ibc75aa98317ca31dbe6a0ffd7bf766732e615f94
Reviewed-on: http://gerrit.cloudera.org:8080/1756
Tested-by: Todd Lipcon <to...@apache.org>
Reviewed-by: Dan Burkert <da...@cloudera.com>
Project: http://git-wip-us.apache.org/repos/asf/incubator-kudu/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-kudu/commit/556bf3b8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-kudu/tree/556bf3b8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-kudu/diff/556bf3b8
Branch: refs/heads/master
Commit: 556bf3b8d9cd8bfd4322456f6e3989bcac528a95
Parents: ad99433
Author: Todd Lipcon <to...@cloudera.com>
Authored: Fri Jan 8 19:30:49 2016 -0800
Committer: Todd Lipcon <to...@apache.org>
Committed: Tue Jan 12 01:49:34 2016 +0000
----------------------------------------------------------------------
build-support/parse_test_failure.py | 13 ++++++++++---
1 file changed, 10 insertions(+), 3 deletions(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/incubator-kudu/blob/556bf3b8/build-support/parse_test_failure.py
----------------------------------------------------------------------
diff --git a/build-support/parse_test_failure.py b/build-support/parse_test_failure.py
index ac96cd8..fd9f8c7 100755
--- a/build-support/parse_test_failure.py
+++ b/build-support/parse_test_failure.py
@@ -109,7 +109,8 @@ def extract_failures(log_text):
record_error(errors_by_test, cur_test_case, error_signature)
# Look for test failures
- m = TEST_FAILURE_RE.search(line)
+ # - slight micro-optimization to check for substring before running the regex
+ m = 'Failure' in line and TEST_FAILURE_RE.search(line)
if m:
error_signature = m.group(0) + "\n"
error_signature += "\n".join(remove_glog_lines(
@@ -117,7 +118,8 @@ def extract_failures(log_text):
record_error(errors_by_test, cur_test_case, error_signature)
# Look for fatal log messages (including CHECK failures)
- m = FATAL_LOG_RE.search(line)
+ # - slight micro-optimization to check for 'F' before running the regex
+ m = line and line[0] == 'F' and FATAL_LOG_RE.search(line)
if m:
error_signature = m.group(1) + "\n"
remaining_lines = consume_rest(line_iter)
@@ -224,9 +226,14 @@ def main():
parser = argparse.ArgumentParser()
parser.add_argument("-x", "--xml", help="Print output in JUnit report XML format (default: plain text)",
action="store_true")
+ parser.add_argument("path", nargs="?", help="File to parse. If not provided, parses stdin")
args = parser.parse_args()
- log_text = sys.stdin.read(MAX_MEMORY)
+ if args.path:
+ in_file = file(args.path)
+ else:
+ in_file = sys.stdin
+ log_text = in_file.read(MAX_MEMORY)
(tests, errors_by_test) = extract_failures(log_text)
print_failure_summary(tests, errors_by_test, args.xml)