You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kudu.apache.org by "Todd Lipcon (Code Review)" <ge...@cloudera.org> on 2016/04/29 20:45:50 UTC
[kudu-CR] WIP: RB to PB Code generation using new flatbuf style codegen
Todd Lipcon has uploaded a new patch set (#2).
Change subject: WIP: RB to PB Code generation using new flatbuf style codegen
......................................................................
WIP: RB to PB Code generation using new flatbuf style codegen
This commit introduces code generation for RowBlock-to-Protobuf conversion.
In addition, this commit unit tests for the RowBlock conversion function, which
make sure that the row-block to protobuf converter produces the same output
that the SerializeRowBlock() function does.
The compilation manager interface is extended and all rowblock-to-pb uses have
been replaced with the codegen version (there's only one, in
tablet_service.cc).
The use of codegen can be toggled with --tablet_service_use_codegen.
Note that this patch also slightly augments cache warmup code in the full stack
test - it now requests a row (and occurs out of the perf-record path). The
wire_protocol-test benchmark was also changed to use codegen when
--wire_protocol_bench_use_codegen is on (on by default).
To test a "real-wife" workload, I ran used tpch_real_world to load 1GB of
lineitem data into a local tablet server, let it fully compact, and then
ran the server with and without codegen enabled:
todd@todd-ThinkPad-T540p:~/git/kudu$ KUDU_HOME=$(pwd) ./build/latest/bin/kudu-tserver -fs_wal_dir /tmp/tpch/ts-0/ -use_hybrid_clock=0 -tablet_service_use_codegen=0 &
todd@todd-ThinkPad-T540p:~/git/kudu$ sudo perf record -a ./build/release/bin/tpch_real_world --tpch_load_data=0 -tpch_test_runtime_sec 60 --tpch_use_mini_cluster=0
I0429 11:37:44.753859 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 2.260s user 0.108s sys 0.004s
I0429 11:37:45.477809 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.724s user 0.084s sys 0.012s
I0429 11:37:46.222630 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.745s user 0.100s sys 0.000s
I0429 11:37:46.946292 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.724s user 0.100s sys 0.000s
I0429 11:37:47.685034 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.739s user 0.096s sys 0.004s
I0429 11:37:48.419046 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.734s user 0.096s sys 0.008s
I0429 11:37:49.169801 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.751s user 0.096s sys 0.008s
I0429 11:37:49.923615 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.754s user 0.104s sys 0.000s
I0429 11:37:50.658107 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.734s user 0.092s sys 0.004s
I0429 11:37:51.384771 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.727s user 0.096s sys 0.000s
I0429 11:37:52.107229 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.722s user 0.080s sys 0.012s
I0429 11:37:52.868758 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.761s user 0.092s sys 0.012s
I0429 11:37:53.622433 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.754s user 0.088s sys 0.012s
I0429 11:37:54.359201 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.737s user 0.096s sys 0.000s
I0429 11:37:55.109233 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.750s user 0.092s sys 0.008s
I0429 11:37:55.848371 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.739s user 0.096s sys 0.004s
I0429 11:37:56.583219 26538 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.735s user 0.088s sys 0.008s
^C[ perf record: Woken up 14 times to write data ]
[ perf record: Captured and wrote 6.035 MB perf.data (81767 samples) ]
todd@todd-ThinkPad-T540p:~/git/kudu$ KUDU_HOME=$(pwd) ./build/latest/bin/kudu-tserver -fs_wal_dir /tmp/tpch/ts-0/ -use_hybrid_clock=0 -tablet_service_use_codegen=1 &
todd@todd-ThinkPad-T540p:~/git/kudu$ sudo perf record -a ./build/release/bin/tpch_real_world --tpch_load_data=0 -tpch_test_runtime_sec 60 --tpch_use_mini_cluster=0
I0429 11:38:18.805004 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 1.960s user 0.112s sys 0.000s
I0429 11:38:19.447537 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.642s user 0.096s sys 0.004s
I0429 11:38:20.062830 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.615s user 0.100s sys 0.004s
I0429 11:38:20.691201 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.628s user 0.092s sys 0.004s
I0429 11:38:21.338073 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.647s user 0.092s sys 0.004s
I0429 11:38:21.975968 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.638s user 0.100s sys 0.004s
I0429 11:38:22.614274 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.638s user 0.096s sys 0.008s
I0429 11:38:23.250212 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.636s user 0.092s sys 0.004s
I0429 11:38:23.885974 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.636s user 0.092s sys 0.008s
I0429 11:38:24.536670 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.651s user 0.092s sys 0.008s
I0429 11:38:25.177048 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.640s user 0.084s sys 0.012s
I0429 11:38:25.824795 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.648s user 0.088s sys 0.004s
I0429 11:38:26.476267 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.651s user 0.092s sys 0.004s
I0429 11:38:27.123643 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.647s user 0.092s sys 0.000s
I0429 11:38:27.759349 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.636s user 0.088s sys 0.004s
I0429 11:38:28.407531 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.648s user 0.096s sys 0.004s
I0429 11:38:29.052520 26706 tpch_real_world.cc:307] Time spent querying data in cluster: real 0.645s user 0.096s sys 0.000s
The average time reduced from about 0.73 seconds to 0.64 seconds (about 15%). Sure
enough, 'perf diff' between the two shows the following difference:
15.81% -15.60% kudu-tserver [.] _ZN4kudu17SerializeRowBlockERKNS_8RowBlockEPNS_17RowwiseRowBlockPBEPKNS_6SchemaEPNS_10faststringES9_
6.19% +0.64% kudu-tserver [.] _ZN4kudu5cfile23BinaryPlainBlockDecoder14CopyNextValuesEPmPNS_14ColumnDataViewE
5.11% +0.59% kudu-tserver [.] _ZN4kudu17DerivedTypeTraitsILNS_8DataTypeE12EE7CompareEPKvS4_
4.09% +0.62% kudu-tserver [.] _ZN4kudu5cfile23BinaryPlainBlockDecoder11ParseHeaderEv
2.34% +0.41% kudu-tserver [.] _ZN4kudu21GetMemcmpableVarint64EPNS_5SliceEPm
1.99% +0.44% kudu-tserver [.] _ZN4kudu6tablet8DeltaKey10DecodeFromEPNS_5SliceE
1.68% -1.23% kudu-tserver [.] _ZN4kudu15BitmapFindFirstEPKhmmbPm
1.67% -0.01% kudu-tserver [.] _ZNK4kudu15ColumnPredicate8EvaluateERKNS_11ColumnBlockEPNS_15SelectionVectorE
1.11% +0.14% kudu-tserver [.] _ZN4kudu6tablet17DeltaFileIterator14VisitMutationsINS0_15ApplyingVisitorILNS0_9DeltaTypeE1EEEEENS_6StatusEPT_
0.76% +0.10% kudu-tserver [.] _ZN4kudu5cfile13CFileIterator12PrepareBatchEPm
0.60% -0.56% kudu-tserver [.] _Z20Hash32StringWithSeedPKcjj
showing that the conversion is no longer a hot spot.
Change-Id: I08db00a8a74653a47261060535a0a316a3e963b0
---
M src/kudu/codegen/CMakeLists.txt
M src/kudu/codegen/code_generator.cc
M src/kudu/codegen/code_generator.h
M src/kudu/codegen/codegen-test.cc
M src/kudu/codegen/codegen_params.fbs
M src/kudu/codegen/compilation_manager.cc
M src/kudu/codegen/compilation_manager.h
M src/kudu/codegen/jit_wrapper.h
M src/kudu/codegen/precompiled.cc
M src/kudu/codegen/row_projector.cc
A src/kudu/codegen/rowblock_converter.cc
A src/kudu/codegen/rowblock_converter.h
M src/kudu/common/CMakeLists.txt
M src/kudu/common/columnblock.h
M src/kudu/common/rowblock.h
M src/kudu/common/wire_protocol-test.cc
M src/kudu/common/wire_protocol.cc
M src/kudu/integration-tests/full_stack-insert-scan-test.cc
M src/kudu/tserver/CMakeLists.txt
M src/kudu/tserver/tablet_service.cc
M src/kudu/util/hdr_histogram.h
21 files changed, 914 insertions(+), 110 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/49/949/2
--
To view, visit http://gerrit.cloudera.org:8080/949
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I08db00a8a74653a47261060535a0a316a3e963b0
Gerrit-PatchSet: 2
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-Owner: Todd Lipcon <to...@apache.org>