You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Joe McDonnell (Jira)" <ji...@apache.org> on 2020/10/07 23:09:00 UTC

[jira] [Updated] (IMPALA-10226) Change buildall.sh -notests to invoke a single Make target

     [ https://issues.apache.org/jira/browse/IMPALA-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe McDonnell updated IMPALA-10226:
-----------------------------------
    Description: 
Currently, running "buildall.sh -notests" boils down to invoking make with multiple targets:

 
{noformat}
  if [[ $BUILD_TESTS -eq 0 ]]; then
    # Specify all the non-test targets
    MAKE_TARGETS="impalad statestored catalogd fesupport loggingsupport ImpalaUdf \
        udasample udfsample"
    if (( build_independent_targets )); then
      MAKE_TARGETS+=" cscope fe tarballs"
    fi
  fi
  ${MAKE_CMD} -j${IMPALA_BUILD_THREADS:-4} ${IMPALA_MAKE_FLAGS} ${MAKE_TARGETS}{noformat}
Based on the build output, it looks like each make target is invoked individually (with the commands underneath going parallel). This is particularly a problem for impalad (which needs to build the backend) and fe. We want these to run simultaneously, and this limitation prevents that.

We should create a single target that builds all the things needing to be built for -notests. Then, this will be invoking one target and allowing all the pieces go parallel.

 

  was:
Currently, running "buildall.sh -notests" boils down to invoking make with multiple targets:

 
{noformat}
  if [[ $BUILD_TESTS -eq 0 ]]; then
    # Specify all the non-test targets
    MAKE_TARGETS="impalad statestored catalogd fesupport loggingsupport ImpalaUdf \
        udasample udfsample"
    if (( build_independent_targets )); then
      MAKE_TARGETS+=" cscope fe tarballs"
    fi
  fi
  ${MAKE_CMD} -j${IMPALA_BUILD_THREADS:-4} ${IMPALA_MAKE_FLAGS} ${MAKE_TARGETS}{noformat}
CMake's Make generator adds the .NOTPARALLEL target to the root Makefile. This means that each make target is invoked serially (while going parallel with the commands underneath going parallel). This is particularly a problem for impalad (which needs to build the backend) and fe. We want these to run simultaneously, and this limitation prevents that.

We should create a single target that builds all the things needing to be built for -notests. Then, this will be invoking one target and allowing all the pieces go parallel.

 


> Change buildall.sh -notests to invoke a single Make target
> ----------------------------------------------------------
>
>                 Key: IMPALA-10226
>                 URL: https://issues.apache.org/jira/browse/IMPALA-10226
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Infrastructure
>    Affects Versions: Impala 4.0
>            Reporter: Joe McDonnell
>            Priority: Major
>
> Currently, running "buildall.sh -notests" boils down to invoking make with multiple targets:
>  
> {noformat}
>   if [[ $BUILD_TESTS -eq 0 ]]; then
>     # Specify all the non-test targets
>     MAKE_TARGETS="impalad statestored catalogd fesupport loggingsupport ImpalaUdf \
>         udasample udfsample"
>     if (( build_independent_targets )); then
>       MAKE_TARGETS+=" cscope fe tarballs"
>     fi
>   fi
>   ${MAKE_CMD} -j${IMPALA_BUILD_THREADS:-4} ${IMPALA_MAKE_FLAGS} ${MAKE_TARGETS}{noformat}
> Based on the build output, it looks like each make target is invoked individually (with the commands underneath going parallel). This is particularly a problem for impalad (which needs to build the backend) and fe. We want these to run simultaneously, and this limitation prevents that.
> We should create a single target that builds all the things needing to be built for -notests. Then, this will be invoking one target and allowing all the pieces go parallel.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org