[jira] [Created] (IMPALA-11623) Put *-ir.cc files into their own libraries to avoid extra recompilation

["Joe McDonnell (Jira)" <ji...@apache.org>]

Joe McDonnell created IMPALA-11623:
--------------------------------------

             Summary: Put *-ir.cc files into their own libraries to avoid extra recompilation
                 Key: IMPALA-11623
                 URL: https://issues.apache.org/jira/browse/IMPALA-11623
             Project: IMPALA
          Issue Type: Improvement
          Components: Infrastructure
    Affects Versions: Impala 4.2.0
            Reporter: Joe McDonnell


It is desirable to be able to iterate quickly by running "make -j impalad" while modifying a file. Currently, modifying most files incurs a rebuild of the LLVM IR, which is a slow serial step. For example:

 
{noformat}
$ touch be/src/runtime/coordinator.cc
$ make -j impalad
...
[ 98%] Generating ../../../llvm-ir/impala.bc
[ 98%] Generating ../../../llvm-ir/impala-legacy-avx.bc
[ 98%] Generating ../../generated-sources/impala-ir/impala-ir.cc
[ 98%] Generating ../../generated-sources/impala-ir/impala-ir-legacy-avx.cc
...{noformat}
This can add several seconds to an incremental build. This step happens for files that do not actually impact the LLVM IR, so there are ways to avoid this.

The reason that LLVM IR is rebuilt is because it has a dependencies on Exec, Exprs, Runtime, Udf, Util, and other libraries:

 
{noformat}
add_custom_command(
  OUTPUT ${IR_OUTPUT_FILE}
  COMMAND ${LLVM_CLANG_EXECUTABLE} ${CLANG_IR_CXX_FLAGS} ${PLATFORM_SPECIFIC_FLAGS}
          ${CLANG_INCLUDE_FLAGS} ${IR_INPUT_FILES} -o ${IR_TMP_OUTPUT_FILE}
  COMMAND ${LLVM_OPT_EXECUTABLE} ${LLVM_OPT_IR_FLAGS} < ${IR_TMP_OUTPUT_FILE} > ${IR_OUTPUT_FILE}
  COMMAND rm ${IR_TMP_OUTPUT_FILE}
  DEPENDS Exec ExecAvro ExecKudu Exprs Runtime Udf Util ${IR_INPUT_FILES}
){noformat}
From a correctness perspective, the LLVM IR only cares about things that impact the content of the *-ir.cc files, because impala-ir.cc includes every *-ir.cc file. That list of libraries is a superset of what is needed.

If the *-ir.cc files were split off into their own libraries (i.e. ExecIr rather than Exec), then this target would only depend on the ExecIr rather than the larger Exec. This would reduce the number of files that would cause LLVM IR to be rebuilt. That should reduce the runtime of an incremental "make -j impalad" for quite a few C++ files.

 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)