You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@maven.apache.org by David Karr <da...@gmail.com> on 2022/03/23 22:11:19 UTC
Strategies for automatically fixing local maven repo corruption?
Our enterprise has builds for a couple of hundred services running many
times a day, on a set of Jenkins build nodes, most of which are Maven
builds.The builds use a settings.xml that doesn't override the
"localRepository", so it will use a local repository on the build node that
is used by all builds.
We often run into weird build errors like "Could not find artifact", for
artifacts that I can clearly show are available in our intranet repository.
In every one of those cases, when we inspect the directory in the local
maven repo on that build node, we see that it's "corrupted", in that
perhaps the pom file is there, but the jar file is missing.
We fix these errors by simply removing the directory where the key files
are missing, then rerun the build, which correctly downloads the artifact,
and the build succeeds. We often have to do this on multiple build nodes,
although I don't have visibility to that (not on ops team), so I don't
really know if the same artifact gets corrupted on multiple build nodes at
the same time. I doubt that.
I wish there was some way that we could either prevent some of these
incidents, or somehow automatically fix them (by removing the corrupted
directory). The latter seems like it's in the realm of possibilities, but
it would have to be constantly scanning the entire repository, looking at
leaf directories that haven't changed in several minutes, and somehow
verifying the "sanity" of the contents from a very limited perspective.
Is there anything we can do to improve this situation?
RE: Strategies for automatically fixing local maven repo corruption?
Posted by Jacques Etienne Beaudet <je...@gmail.com>.
You're right sorry about that https://svn.apache.org/repos/asf/maven/website/components/resolver-archives/resolver-1.6.3/maven-resolver-synccontext-redisson/index.html
On Mar 23, 2022, 6:28 PM -0400, KARR, DAVID <dk...@att.com>, wrote:
> > -----Original Message-----
> > From: Jacques Etienne Beaudet <je...@gmail.com>
> > Sent: Wednesday, March 23, 2022 3:14 PM
> > To: Maven Users List <us...@maven.apache.org>
> > Subject: Re: Strategies for automatically fixing local maven repo
> > corruption?
> >
> > I'll copy paste something I wrote in another thread but yea basically
> > there's a neat way to help with this :
> >
> > Just chiming in on my experience using the redis locks with maven 3.8.x
> > and resolver 1.6. I've been running it on my Jenkins nodes for about a
> > year now with great success. Before it, we had corrupted local maven
> > repos (zip file empty kind of exceptions) a couple of times a week.
> > Then, we configured Jenkins to properly handle multiple concurrent
> > builds by not sharing the maven local repo and run each build in docker
> > which is wasteful since you have to redownload all your dependencies on
> > each build and build time is obviously very long.
> >
> > Then I bumped into the lock feature which is available on resolver 1.6
> > (instructions here svn.apache.org/repos...x.html). It is a bit
> > cumbersome to setup but after that, we started sharing the same local
> > repos and our build time dramatically improved (something like from 4
> > minutes to ~30 seconds) and we had a corrupted maven repo once or twice
> > throughout the year (and it could have been some job that a dev created
> > without the proper config, I didn't investigate much).
>
> This note posted with that url truncated. Can you provide the full url?
>
> > Redis is really easy to setup and is available via docker or apt so it
> > should be a breeze to add it to your init script like we did. My nodes
> > had docker already installed so it was just adding this to the "Init
> > script" in the nodes configuration :
> > #!/bin/bash -ue
> >
> > docker run -d --network host --name mvn_redis redis:6.2-alpine
>
> > 23, 2022, 6:11 PM -0400, David Karr <da...@gmail.com>, wrote:
> > > Our enterprise has builds for a couple of hundred services running
> > > many times a day, on a set of Jenkins build nodes, most of which are
> > > Maven builds.The builds use a settings.xml that doesn't override the
> > > "localRepository", so it will use a local repository on the build node
> > > that is used by all builds.
> > >
> > > We often run into weird build errors like "Could not find artifact",
> > > for artifacts that I can clearly show are available in our intranet
> > repository.
> > > In every one of those cases, when we inspect the directory in the
> > > local maven repo on that build node, we see that it's "corrupted", in
> > > that perhaps the pom file is there, but the jar file is missing.
> > >
> > > We fix these errors by simply removing the directory where the key
> > > files are missing, then rerun the build, which correctly downloads the
> > > artifact, and the build succeeds. We often have to do this on multiple
> > > build nodes, although I don't have visibility to that (not on ops
> > > team), so I don't really know if the same artifact gets corrupted on
> > > multiple build nodes at the same time. I doubt that.
> > >
> > > I wish there was some way that we could either prevent some of these
> > > incidents, or somehow automatically fix them (by removing the
> > > corrupted directory). The latter seems like it's in the realm of
> > > possibilities, but it would have to be constantly scanning the entire
> > > repository, looking at leaf directories that haven't changed in
> > > several minutes, and somehow verifying the "sanity" of the contents
> > from a very limited perspective.
> > >
> > > Is there anything we can do to improve this situation?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@maven.apache.org
> For additional commands, e-mail: users-help@maven.apache.org
RE: Strategies for automatically fixing local maven repo corruption?
Posted by "KARR, DAVID" <dk...@att.com>.
> -----Original Message-----
> From: Jacques Etienne Beaudet <je...@gmail.com>
> Sent: Wednesday, March 23, 2022 3:14 PM
> To: Maven Users List <us...@maven.apache.org>
> Subject: Re: Strategies for automatically fixing local maven repo
> corruption?
>
> I'll copy paste something I wrote in another thread but yea basically
> there's a neat way to help with this :
>
> Just chiming in on my experience using the redis locks with maven 3.8.x
> and resolver 1.6. I've been running it on my Jenkins nodes for about a
> year now with great success. Before it, we had corrupted local maven
> repos (zip file empty kind of exceptions) a couple of times a week.
> Then, we configured Jenkins to properly handle multiple concurrent
> builds by not sharing the maven local repo and run each build in docker
> which is wasteful since you have to redownload all your dependencies on
> each build and build time is obviously very long.
>
> Then I bumped into the lock feature which is available on resolver 1.6
> (instructions here svn.apache.org/repos...x.html). It is a bit
> cumbersome to setup but after that, we started sharing the same local
> repos and our build time dramatically improved (something like from 4
> minutes to ~30 seconds) and we had a corrupted maven repo once or twice
> throughout the year (and it could have been some job that a dev created
> without the proper config, I didn't investigate much).
This note posted with that url truncated. Can you provide the full url?
> Redis is really easy to setup and is available via docker or apt so it
> should be a breeze to add it to your init script like we did. My nodes
> had docker already installed so it was just adding this to the "Init
> script" in the nodes configuration :
> #!/bin/bash -ue
>
> docker run -d --network host --name mvn_redis redis:6.2-alpine
> 23, 2022, 6:11 PM -0400, David Karr <da...@gmail.com>, wrote:
> > Our enterprise has builds for a couple of hundred services running
> > many times a day, on a set of Jenkins build nodes, most of which are
> > Maven builds.The builds use a settings.xml that doesn't override the
> > "localRepository", so it will use a local repository on the build node
> > that is used by all builds.
> >
> > We often run into weird build errors like "Could not find artifact",
> > for artifacts that I can clearly show are available in our intranet
> repository.
> > In every one of those cases, when we inspect the directory in the
> > local maven repo on that build node, we see that it's "corrupted", in
> > that perhaps the pom file is there, but the jar file is missing.
> >
> > We fix these errors by simply removing the directory where the key
> > files are missing, then rerun the build, which correctly downloads the
> > artifact, and the build succeeds. We often have to do this on multiple
> > build nodes, although I don't have visibility to that (not on ops
> > team), so I don't really know if the same artifact gets corrupted on
> > multiple build nodes at the same time. I doubt that.
> >
> > I wish there was some way that we could either prevent some of these
> > incidents, or somehow automatically fix them (by removing the
> > corrupted directory). The latter seems like it's in the realm of
> > possibilities, but it would have to be constantly scanning the entire
> > repository, looking at leaf directories that haven't changed in
> > several minutes, and somehow verifying the "sanity" of the contents
> from a very limited perspective.
> >
> > Is there anything we can do to improve this situation?
Re: Strategies for automatically fixing local maven repo corruption?
Posted by Jacques Etienne Beaudet <je...@gmail.com>.
I'll copy paste something I wrote in another thread but yea basically there's a neat way to help with this :
Just chiming in on my experience using the redis locks with maven 3.8.x and resolver 1.6. I've been running it on my Jenkins nodes for about a year now with great success. Before it, we had corrupted local maven repos (zip file empty kind of exceptions) a couple of times a week. Then, we configured Jenkins to properly handle multiple concurrent builds by not sharing the maven local repo and run each build in docker which is wasteful since you have to redownload all your dependencies on each build and build time is obviously very long.
Then I bumped into the lock feature which is available on resolver 1.6 (instructions here svn.apache.org/repos...x.html). It is a bit cumbersome to setup but after that, we started sharing the same local repos and our build time dramatically improved (something like from 4 minutes to ~30 seconds) and we had a corrupted maven repo once or twice throughout the year (and it could have been some job that a dev created without the proper config, I didn't investigate much).
Redis is really easy to setup and is available via docker or apt so it should be a breeze to add it to your init script like we did. My nodes had docker already installed so it was just adding this to the "Init script" in the nodes configuration :
#!/bin/bash -ue
docker run -d --network host --name mvn_redis redis:6.2-alpine
On Mar 23, 2022, 6:11 PM -0400, David Karr <da...@gmail.com>, wrote:
> Our enterprise has builds for a couple of hundred services running many
> times a day, on a set of Jenkins build nodes, most of which are Maven
> builds.The builds use a settings.xml that doesn't override the
> "localRepository", so it will use a local repository on the build node that
> is used by all builds.
>
> We often run into weird build errors like "Could not find artifact", for
> artifacts that I can clearly show are available in our intranet repository.
> In every one of those cases, when we inspect the directory in the local
> maven repo on that build node, we see that it's "corrupted", in that
> perhaps the pom file is there, but the jar file is missing.
>
> We fix these errors by simply removing the directory where the key files
> are missing, then rerun the build, which correctly downloads the artifact,
> and the build succeeds. We often have to do this on multiple build nodes,
> although I don't have visibility to that (not on ops team), so I don't
> really know if the same artifact gets corrupted on multiple build nodes at
> the same time. I doubt that.
>
> I wish there was some way that we could either prevent some of these
> incidents, or somehow automatically fix them (by removing the corrupted
> directory). The latter seems like it's in the realm of possibilities, but
> it would have to be constantly scanning the entire repository, looking at
> leaf directories that haven't changed in several minutes, and somehow
> verifying the "sanity" of the contents from a very limited perspective.
>
> Is there anything we can do to improve this situation?
Re: Strategies for automatically fixing local maven repo corruption?
Posted by Bernd Eckenfels <ec...@zusammenkunft.net>.
You should avoid concurrent builds on a local Repo. Either use a “per worker” Repo (all good CI servers should support that, but you can also do it by hand command line option) or use a Container build process.
There is also a locking scheme for Maven now, but I would start with Seperate repos.
And also clean them up at least once a week or more often, that also helps against other corruptions and reproducibility issues.
You always should have a caching mirror to make this less impactful.
Gruss
Bernd
--
http://bernd.eckenfels.net
________________________________
Von: David Karr <da...@gmail.com>
Gesendet: Wednesday, March 23, 2022 11:11:19 PM
An: Maven Users List <us...@maven.apache.org>
Betreff: Strategies for automatically fixing local maven repo corruption?
Our enterprise has builds for a couple of hundred services running many
times a day, on a set of Jenkins build nodes, most of which are Maven
builds.The builds use a settings.xml that doesn't override the
"localRepository", so it will use a local repository on the build node that
is used by all builds.
We often run into weird build errors like "Could not find artifact", for
artifacts that I can clearly show are available in our intranet repository.
In every one of those cases, when we inspect the directory in the local
maven repo on that build node, we see that it's "corrupted", in that
perhaps the pom file is there, but the jar file is missing.
We fix these errors by simply removing the directory where the key files
are missing, then rerun the build, which correctly downloads the artifact,
and the build succeeds. We often have to do this on multiple build nodes,
although I don't have visibility to that (not on ops team), so I don't
really know if the same artifact gets corrupted on multiple build nodes at
the same time. I doubt that.
I wish there was some way that we could either prevent some of these
incidents, or somehow automatically fix them (by removing the corrupted
directory). The latter seems like it's in the realm of possibilities, but
it would have to be constantly scanning the entire repository, looking at
leaf directories that haven't changed in several minutes, and somehow
verifying the "sanity" of the contents from a very limited perspective.
Is there anything we can do to improve this situation?