You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2020/06/12 01:55:00 UTC
[jira] [Commented] (IMPALA-9107) Reduce time spent downloading
maven artifacts for precommit tests
[ https://issues.apache.org/jira/browse/IMPALA-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17133834#comment-17133834 ]
ASF subversion and git services commented on IMPALA-9107:
---------------------------------------------------------
Commit fb282852ef52d72079a86c55a90982ffac567cc7 in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=fb28285 ]
IMPALA-9107 (part 2): Add script to use the m2 archive tarball
This adds a script to find an appropriate m2 archive
tarball, download it, and use it to prepopulate the
~/.m2 directory.
The script uses the JSON interface for Jenkins to search through
the all-build-options-ub1604 builds on jenkins.impala.io to
find one that:
1. Is building the "master" branch
2. Has the m2_archive.tar.gz
Then, it downloads the m2 archive and uses it to populate ~/.m2.
It does not overwrite or remove any files already in ~/.m2.
The build scripts that call populate_m2_directory.py do not
rely on the script succeeding. They will continue even if
the script fails.
This also modifies the build-all-flag-combinations.sh script
to only build the m2 archive if the GENERATE_M2_ARCHIVE
environment variable is true. GENERATE_M2_ARCHIVE=true will
clear out the ~/.m2 directory to build an accurate m2 archive.
Precommit jobs will use GENERATE_M2_ARCHIVE=false, which
will allow them to use the m2 archive to speed up the build.
Testing:
- Ran gerrify-verify-dryrun
- Tested locally
Change-Id: I5065658d8c0514550927161855b0943fa7b3a402
Reviewed-on: http://gerrit.cloudera.org:8080/15735
Reviewed-by: Joe McDonnell <jo...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
> Reduce time spent downloading maven artifacts for precommit tests
> -----------------------------------------------------------------
>
> Key: IMPALA-9107
> URL: https://issues.apache.org/jira/browse/IMPALA-9107
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend, Infrastructure
> Affects Versions: Impala 3.4.0
> Reporter: Joe McDonnell
> Priority: Major
>
> When building Impala from scratch with an empty .m2 directory, maven needs to download a large number of jars and pom files. This is taking a long time and adds about 15-20 minutes to the build. For example, here is some output from a recent ubuntu-16.04-from-scratch, there is an 18 minute delay between the end of building the backend tests and the end of building the frontend:
> {code:java}
> 00:58:33 [100%] Built target hash-ring-util
> ...
> 01:16:37 [100%] Built target fe{code}
> Almost all of that time is spent downloading maven artifacts. Quite a few come from the maven central repository.
> This is taking way too much time, and we need to reduce it. The total size of artifacts being downloaded is not large. One approach would be to produce a tarball with the jars/poms that don't come from the CDP or CDH GBN repos. We can download that tarball and use it to either populate the .m2 directory or to stash it in IMPALA_TOOLCHAIN and use it as a maven repository with a file:// URI.
> This impacts all our jobs: ubuntu-16.04-from-scratch, ubuntu-16.04-dockerised-tests, all-build-options-ub1604, ubuntu-16.04-build-only, clang-tidy-ub1604
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org