[Impala-CR](cdh5-trunk) Split LLVM bitcode into multiple modules

["Tim Armstrong (Code Review)" <ge...@cloudera.org>]

Tim Armstrong has uploaded a new patch set (#3).

Change subject: Split LLVM bitcode into multiple modules
......................................................................

Split LLVM bitcode into multiple modules

Each -ir.cc file is compiled into a separate bitcode file. When a query
is codegened, only the required modules are linked into a main module.
We determine which modules to link based on which functions are looked
up: modules are linked into the main module as functions are required.

One tricky aspect is getting the IR types to be merged together and
to be assigned the with the names that we expect. We force this by
prepopulating the main module with the expected types, so that they
will have their canonical names. This doesn't work for the recursive
FunctionContext type. In that special case we just need to add bitcasts.

The patch includes a mechanism to resolve dependencies between modules:
in several cases we have IR functions that call IR functions in other
modules under the assumption that they will end up in the same IR
module. E.g. FunctionContext::GetFunctionState() and
DecimalOperators::Round() are both called from functions in other IR
modules.

Performance:
Module prepare time is dramatically reduced for typical fragments that
use only a small subset of the cross-compiled IR. E.g. for the query
select count(*) from tpch_parquet.lineitem that only uses the
IR from partitioned-aggregation-node-ir.cc, codegen time for the
first fragment is reduced by ~115ms or 95%. PrepareTime is reduced
because less bitcode needs to be parsed, and OptimizationTime is reduced
because less dead code needs to be eliminated.

select count(*) before:
      CodeGen:(Total: 124.424ms, non-child: 124.424ms, % non-child: 100.00%)
         - CodegenTime: 187.740us
         - CompileTime: 2.109ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 2.19 MB (2294104)
         - NumFunctions: 4 (4)
         - NumInstructions: 56 (56)
         - OptimizationTime: 23.311ms
         - PrepareTime: 98.727ms

select count(*) after:
      CodeGen:(Total: 7.090ms, non-child: 7.090ms, % non-child: 100.00%)
         - CodegenTime: 2.411ms
         - CompileTime: 1.536ms
         - LoadTime: 0.000ns
         - ModuleBitcodeSize: 114.37 KB (117112)
         - NumFunctions: 4 (4)
         - NumInstructions: 56 (56)
         - OptimizationTime: 2.377ms
         - PrepareTime: 2.066ms

More complex plan fragments show less benefit. E.g. for a fragment in
TPC-H Q14 with an agg and join, codegen time is reduced by around 50ms
or ~10%. The benefit is less because it pulls in some larger bitcode
modules. A follow-up patch addresses this partially by breaking up some
of the larger bitcode modules.

Change-Id: Ib48ea5276d3f2c0752a7e37d6037ce388769f753
---
M be/src/benchmarks/atod-benchmark.cc
M be/src/codegen/CMakeLists.txt
M be/src/codegen/codegen-anyval.cc
A be/src/codegen/codegen-util.cc
A be/src/codegen/codegen-util.h
M be/src/codegen/gen_ir_descriptions.py
D be/src/codegen/impala-ir-data.h
D be/src/codegen/impala-ir.cc
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
A be/src/codegen/main-module-ir.cc
M be/src/exec/aggregation-node-ir.cc
M be/src/exec/aggregation-node.cc
M be/src/exec/hash-join-node.cc
M be/src/exec/hash-table.cc
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/partitioned-aggregation-node.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/text-converter.cc
M be/src/exprs/CMakeLists.txt
M be/src/exprs/anyval-util.h
M be/src/exprs/expr.cc
M be/src/exprs/in-predicate-benchmark.cc
M be/src/exprs/scalar-fn-call.cc
M be/src/util/sse-util.h
M be/src/util/symbols-util-test.cc
28 files changed, 666 insertions(+), 260 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/24/2924/3
-- 
To view, visit http://gerrit.cloudera.org:8080/2924
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib48ea5276d3f2c0752a7e37d6037ce388769f753
Gerrit-PatchSet: 3
Gerrit-Project: Impala
Gerrit-Branch: cdh5-trunk
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>