Re: [EXTERNAL] Improving branch-3 Build Times

[Aman Raj <ra...@microsoft.com.INVALID>]

This looks great Chris. You can raise a PR and we can start discussions on the same.

Thanks,
Aman.
________________________________
From: Chris Nauroth <cn...@apache.org>
Sent: Saturday, January 7, 2023 5:29 AM
To: dev <de...@hive.apache.org>
Subject: [EXTERNAL] Improving branch-3 Build Times

For those of you working on both master and branch-3, you may have noticed
that it takes a crazy long time to complete a full build on branch-3. In my
environment, running this...

mvn -B -T 8 -Pitests clean install -DskipTests

...it completes in ~8 minutes on master, but ~40 minutes on branch-3. The
long haul is modules like standalone-metastore that use
maven-assembly-plugin to build the artifact. At least part of the root
cause is that branch-3 is using an older version of maven-assembly-plugin
with sub-optimal file system access patterns.

I have a work in progress patch here:

https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fhive%2Fpull%2F3924&data=05%7C01%7Crajaman%40microsoft.com%7Caa65b42ed50340db79c808daf042201d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638086464128742622%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8gsYRq5PZdLQBRjkgM8J7eOluNeQz56sKMmGOSjrETI%3D&reserved=0

This is just a draft, not ready for review, but so far I've been able to
cut my branch-3 build times back down to ~13 minutes. I might not be able
to polish this off and request review until late next week, so I'm sharing
it early in case it makes life easier for anyone else on branch-3.

Chris Nauroth