[jira] [Created] (IMPALA-9107) Reduce time spent downloading maven artifacts for precommit tests

["Joe McDonnell (Jira)" <ji...@apache.org>]

Joe McDonnell created IMPALA-9107:
-------------------------------------

             Summary: Reduce time spent downloading maven artifacts for precommit tests
                 Key: IMPALA-9107
                 URL: https://issues.apache.org/jira/browse/IMPALA-9107
             Project: IMPALA
          Issue Type: Improvement
          Components: Frontend, Infrastructure
    Affects Versions: Impala 3.4.0
            Reporter: Joe McDonnell


When building Impala from scratch with an empty .m2 directory, maven needs to download a large number of jars and pom files. This is taking a long time and adds about 15-20 minutes to the build. For example, here is some output from a recent ubuntu-16.04-from-scratch, there is an 18 minute delay between the end of building the backend tests and the end of building the frontend:
{code:java}
00:58:33 [100%] Built target hash-ring-util
...
01:16:37 [100%] Built target fe{code}
Almost all of that time is spent downloading maven artifacts. Quite a few come from the maven central repository.

This is taking way too much time, and we need to reduce it. The total size of artifacts being downloaded is not large. One approach would be to produce a tarball with the jars/poms that don't come from the CDP or CDH GBN repos. We can download that tarball and use it to either populate the .m2 directory or to stash it in IMPALA_TOOLCHAIN and use it as a maven repository with a file:// URI.

This impacts all our jobs: ubuntu-16.04-from-scratch, ubuntu-16.04-dockerised-tests, all-build-options-ub1604, ubuntu-16.04-build-only, clang-tidy-ub1604

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)