Network failures while maven runs

[Luca Toscano <to...@gmail.com>]

Hi everybody,

I have been seeing build failures for trunk packages due to maven
network failures (mostly connection timeouts). Is there a workaround
for this? For example retrying x times etc.. I am asking mostly
because of ignorance, I am wondering what we do when cutting a release
for example (to avoid re-running the main package build job due to
some package build failures).

Thanks!

Luca

Re: Network failures while maven runs

[Luca Toscano <to...@gmail.com>]

Hi Olaf,

On Sat, Jan 29, 2022 at 9:18 AM Olaf Flebbe <of...@oflebbe.de> wrote:
>
> Hi Luca,
>
> I think you are referring to random failures of maven downloads?

Exactly yes, maven download tasks and also random connect timeouts
when downloading poms.
>
> My next proposal would be to use docker volumes (not bind mounts) for that, so we wont have any permission problems any more and we are left with the server issues going out if business, eventually. To mitigate that it would be easy to simply recreate docker volumes rather to go to all filesystems as root removing directories we needed for the bind mount workaround.

To better understand your proposal - the Docker volume would be a
shared maven cache that we'll mount to containers when building? Or
something else?

> We could eventually pair to try to implement that, if you are not too familiar with it.
> https://docs.docker.com/storage/volumes/

Definitely, I'd be happy to! Never used volumes up to now, only bind mounts.

Luca

Re: Network failures while maven runs

[Olaf Flebbe <of...@oflebbe.de>]

Hi Luca,

I think you are referring to random failures of maven downloads?

We used a couple of countermeasures in the past, all of them proved problematic.

We persisted the maven cache so the next by bind mounting a local directory onto the maven cache .m2 . We had nice permission issues and didn‘t triggered when a maven repository went out of business: Our build suceeded but everyone else wasn‘t able to build any more. Since everything now converges to maven central this might not be that big issue any more. 

A different approach was to use a proxy maven repo, i remember we had issues because it somehow didn‘t work, and we have the hurdle of yet another tool to master. 

Both of these worked as a workaround, somehow to cut releases.

On my daytime job the devs configured their job to prepopulate the maven cache , but I am not a fan of this approach.

My next proposal would be to use docker volumes (not bind mounts) for that, so we wont have any permission problems any more and we are left with the server issues going out if business, eventually. To mitigate that it would be easy to simply recreate docker volumes rather to go to all filesystems as root removing directories we needed for the bind mount workaround.

We could eventually pair to try to implement that, if you are not too familiar with it.
https://docs.docker.com/storage/volumes/

Best
Olaf


> Am 29.01.2022 um 08:31 schrieb Luca Toscano <to...@gmail.com>:
> 
> Hi everybody,
> 
> I have been seeing build failures for trunk packages due to maven
> network failures (mostly connection timeouts). Is there a workaround
> for this? For example retrying x times etc.. I am asking mostly
> because of ignorance, I am wondering what we do when cutting a release
> for example (to avoid re-running the main package build job due to
> some package build failures).
> 
> Thanks!
> 
> Luca

Re: Network failures while maven runs

[Luca Toscano <to...@gmail.com>]

Hi Konstantin,

I am seeing two type of failures:

1) Maven download errors that don't show (afaics) any hint about what
failed. Example:
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=kafka,OS=centos-7-ppc64le/lastBuild/console

2) Connect timeouts while downloading poms, example:
https://ci.bigtop.apache.org/view/Packages/job/Bigtop-trunk-packages/COMPONENTS=hive,OS=debian-10-ppc64le/lastBuild/console

Usually 2) seems to be related to apache.org-related repo servers.

I am not very expert with Maven but I was wondering how Bigtop
releases were cut with such a degree of unpredictability in builds (if
some secret option was used etc..). I naively thought that adding a
basic retry to Maven would have improved the end result, but over time
a lot of people (like Olaf mentioned) tried to resolve this problem so
I preferred to ask first, before making silly suggestions :)

Thanks,

Luca

On Mon, Jan 31, 2022 at 2:21 PM Konstantin Boudnik <co...@apache.org> wrote:
>
> Hey Luca.
>
> Did you notice if this is happening with some specific repo server?
>
> Tahnks!
>   Cos
>
> On Sat, Jan 29, 2022 at 08:31AM, Luca Toscano wrote:
> > Hi everybody,
> >
> > I have been seeing build failures for trunk packages due to maven
> > network failures (mostly connection timeouts). Is there a workaround
> > for this? For example retrying x times etc.. I am asking mostly
> > because of ignorance, I am wondering what we do when cutting a release
> > for example (to avoid re-running the main package build job due to
> > some package build failures).
> >
> > Thanks!
> >
> > Luca

Re: Network failures while maven runs

[Konstantin Boudnik <co...@apache.org>]

Hey Luca.

Did you notice if this is happening with some specific repo server?

Tahnks!
  Cos

On Sat, Jan 29, 2022 at 08:31AM, Luca Toscano wrote:
> Hi everybody,
> 
> I have been seeing build failures for trunk packages due to maven
> network failures (mostly connection timeouts). Is there a workaround
> for this? For example retrying x times etc.. I am asking mostly
> because of ignorance, I am wondering what we do when cutting a release
> for example (to avoid re-running the main package build job due to
> some package build failures).
> 
> Thanks!
> 
> Luca