[jira] [Commented] (IMPALA-9107) Reduce time spent downloading maven artifacts for precommit tests

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17065139#comment-17065139 ] 

ASF subversion and git services commented on IMPALA-9107:
---------------------------------------------------------

Commit 603e5147d59a4f6a79e5cfeaf9d1421d01bf2d09 in impala's branch refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=603e514 ]

IMPALA-9107 (part 1): Add scripts to produce an m2 archive

The maven build downloads a large number of artifacts from
various maven repositories. When starting with an empty .m2
directory (like most upstream Jenkins jobs), downloading
all the artifacts can take up to 30 minutes. This has been
slowing down our precommit builds by 15-20 minutes.

This adds a script to archive the .m2 directory into a
tarball while excluding artifacts from impala.cdp.repo
and impala.cdh.repo. This will later be used to prepopulate
the .m2 directory for Jenkins jobs.

This adds a script to parse the maven log and output how
many maven artifacts are downloaded from each repository.
It also prints how many downloads were attempted for each
repository. This might aid in diagnosing slowness.

This also changes mvn-quiet.sh to add logging that prints
a timestamp. It also adds the -B flag to mvn, which causes
maven to run in batch mode. This makes the output easier
to parse, because maven omits special console formatting
characters such as ^M (carriange return).

This changes build-all-flag-combinations.sh to print the
maven statistics after each part of the build and call the
script to produce an m2 archive at the end.

Change-Id: I043912f5fbc7cf24ee80b2855354656aa587ca9f
Reviewed-on: http://gerrit.cloudera.org:8080/14562
Reviewed-by: Laszlo Gaal <la...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Reduce time spent downloading maven artifacts for precommit tests
> -----------------------------------------------------------------
>
>                 Key: IMPALA-9107
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9107
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Frontend, Infrastructure
>    Affects Versions: Impala 3.4.0
>            Reporter: Joe McDonnell
>            Priority: Major
>
> When building Impala from scratch with an empty .m2 directory, maven needs to download a large number of jars and pom files. This is taking a long time and adds about 15-20 minutes to the build. For example, here is some output from a recent ubuntu-16.04-from-scratch, there is an 18 minute delay between the end of building the backend tests and the end of building the frontend:
> {code:java}
> 00:58:33 [100%] Built target hash-ring-util
> ...
> 01:16:37 [100%] Built target fe{code}
> Almost all of that time is spent downloading maven artifacts. Quite a few come from the maven central repository.
> This is taking way too much time, and we need to reduce it. The total size of artifacts being downloaded is not large. One approach would be to produce a tarball with the jars/poms that don't come from the CDP or CDH GBN repos. We can download that tarball and use it to either populate the .m2 directory or to stash it in IMPALA_TOOLCHAIN and use it as a maven repository with a file:// URI.
> This impacts all our jobs: ubuntu-16.04-from-scratch, ubuntu-16.04-dockerised-tests, all-build-options-ub1604, ubuntu-16.04-build-only, clang-tidy-ub1604
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org