You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tomee.apache.org by Cesar Hernandez <ce...@gmail.com> on 2022/04/26 17:39:07 UTC
Increasing Discard Old Builds strategy in CI
Hello,
Our current Discard old builds Strategy in CI for PR's for the master
branch is [1]:
Days to keep builds: 2
Max # of builds to keep: 3
Is there any objection if I update this to at least:
Days to keep builds: 15
Max # of builds to keep: 30
Currently, if one opens a PR on a Friday and tries to catch up on
mid-Monday next week, the PR results are gone.
The above proposal will also give us a bit more room for more context on
how PR's and past builds have evolved, especially when trying to chase and
resolve random test failures.
[1]
https://ci-builds.apache.org/job/Tomee/job/master-pull-request
--
Atentamente:
César Hernández.
Re: Increasing Discard Old Builds strategy in CI
Posted by Cesar Hernandez <ce...@gmail.com>.
Thank you for the context Richard,
My proposal is for master PR's only since I think this is where most of the
activity is going on at the moment.
We occupied 1,5T last week
In that case, I think a more conservative proposal would be:
Days to keep builds: 8
Max # of builds to keep: 15
El mar, 26 abr 2022 a las 11:47, Zowalla, Richard (<
richard.zowalla@hs-heilbronn.de>) escribió:
> Hi Cesar,
>
> I have no problem with increasing the values for the discard strategy.
> It makes sense to keep it a bit longer for PRs.
>
> However, we need to carefully monitor the disk usage / volume used as
> INFRA has a problem if our used storage volume per jobs exceeds a
> certain quota.
>
> We occupied 1,5T last week (cumulated over a few weeks) and they were a
> bit mad as the Jenkins CI env went out of storage.
>
> Gruß
> Richard
>
>
> Am Dienstag, dem 26.04.2022 um 11:39 -0600 schrieb Cesar Hernandez:
> > Hello,
> >
> > Our current Discard old builds Strategy in CI for PR's for the master
> > branch is [1]:
> > Days to keep builds: 2
> > Max # of builds to keep: 3
> >
> > Is there any objection if I update this to at least:
> > Days to keep builds: 15
> > Max # of builds to keep: 30
> >
> > Currently, if one opens a PR on a Friday and tries to catch up on
> > mid-Monday next week, the PR results are gone.
> > The above proposal will also give us a bit more room for more context
> > on
> > how PR's and past builds have evolved, especially when trying to
> > chase and
> > resolve random test failures.
> >
> >
> > [1]
> > https://ci-builds.apache.org/job/Tomee/job/master-pull-request
>
--
Atentamente:
César Hernández.
Re: Increasing Discard Old Builds strategy in CI
Posted by Cesar Hernandez <ce...@gmail.com>.
Hello,
I found the post-action "Delete Workspace when build is done", I updated
https://ci-builds.apache.org/job/TomEE/job/master-pull-request/ to check if
this can reduce the job size.
If this doesn't work I'll rollback the change and keep you updated.
[image: image.png]
El mar, 26 abr 2022 a las 13:43, Cesar Hernandez (<ce...@gmail.com>)
escribió:
> I forgot that /home/jenkins/.m2 is in the Agent and what is actually
> analyzed for the test results are the surefire files as Richard mentioned:
>
> /home/jenkins/jenkins-agent/workspace/Tomee/master-pull-request/
>
>
> I'm +1 on trying a mvn clean as a post step to see if that reduces the
> size of the job result without loosing logs.
>
>
>
>
> El mar, 26 abr 2022 a las 12:52, Cesar Hernandez (<ce...@gmail.com>)
> escribió:
>
>> In the job, after the test results are executed we can add a post-build
>> doing a:
>> rm -rf /home/jenkins/.m2/
>>
>>
>>
>>
>> El mar, 26 abr 2022 a las 12:51, Zowalla, Richard (<
>> richard.zowalla@hs-heilbronn.de>) escribió:
>>
>>> Afaik Jenkins is parsing the surefire report files. The log files might
>>> also be important for the creator of the PR, so we would need to keep
>>> them too.
>>>
>>> Might be worth a try, I guess.
>>>
>>> Gruß
>>> Richard
>>>
>>> Am Dienstag, dem 26.04.2022 um 11:15 -0700 schrieb David Blevins:
>>> > > On Apr 26, 2022, at 10:47 AM, Zowalla, Richard <
>>> > > richard.zowalla@hs-heilbronn.de> wrote:
>>> > >
>>> > > However, we need to carefully monitor the disk usage / volume used
>>> > > as
>>> > > INFRA has a problem if our used storage volume per jobs exceeds a
>>> > > certain quota.
>>> > >
>>> > > We occupied 1,5T last week (cumulated over a few weeks) and they
>>> > > were a
>>> > > bit mad as the Jenkins CI env went out of storage.
>>> >
>>> > I was going to jump in on that thread and got side tracked.
>>> >
>>> > I wonder if we can do a `mvn clean` or something similar as a post
>>> > build step. Looks like after a build we have 2.8G used of disk space
>>> > and once we clean that goes down to 398M.
>>> >
>>> > The test case results are the obvious trick. I'm not sure if Jenkins
>>> > keeps those somewhere else or if we need them to keep living in the
>>> > target directories. If they need to stay in the target directories
>>> > we can maybe create a script that deletes everything in target/ but
>>> > surefire results.
>>> >
>>> > If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5
>>> > gigabytes.
>>> >
>>> > Then we could just keep 30 builds regardless of days and it should be
>>> > good.
>>> >
>>> > Thoughts?
>>> >
>>> >
>>> > -David
>>> >
>>>
>>
>>
>> --
>> Atentamente:
>> César Hernández.
>>
>
>
> --
> Atentamente:
> César Hernández.
>
--
Atentamente:
César Hernández.
Re: Increasing Discard Old Builds strategy in CI
Posted by Cesar Hernandez <ce...@gmail.com>.
I forgot that /home/jenkins/.m2 is in the Agent and what is actually
analyzed for the test results are the surefire files as Richard mentioned:
/home/jenkins/jenkins-agent/workspace/Tomee/master-pull-request/
I'm +1 on trying a mvn clean as a post step to see if that reduces the size
of the job result without loosing logs.
El mar, 26 abr 2022 a las 12:52, Cesar Hernandez (<ce...@gmail.com>)
escribió:
> In the job, after the test results are executed we can add a post-build
> doing a:
> rm -rf /home/jenkins/.m2/
>
>
>
>
> El mar, 26 abr 2022 a las 12:51, Zowalla, Richard (<
> richard.zowalla@hs-heilbronn.de>) escribió:
>
>> Afaik Jenkins is parsing the surefire report files. The log files might
>> also be important for the creator of the PR, so we would need to keep
>> them too.
>>
>> Might be worth a try, I guess.
>>
>> Gruß
>> Richard
>>
>> Am Dienstag, dem 26.04.2022 um 11:15 -0700 schrieb David Blevins:
>> > > On Apr 26, 2022, at 10:47 AM, Zowalla, Richard <
>> > > richard.zowalla@hs-heilbronn.de> wrote:
>> > >
>> > > However, we need to carefully monitor the disk usage / volume used
>> > > as
>> > > INFRA has a problem if our used storage volume per jobs exceeds a
>> > > certain quota.
>> > >
>> > > We occupied 1,5T last week (cumulated over a few weeks) and they
>> > > were a
>> > > bit mad as the Jenkins CI env went out of storage.
>> >
>> > I was going to jump in on that thread and got side tracked.
>> >
>> > I wonder if we can do a `mvn clean` or something similar as a post
>> > build step. Looks like after a build we have 2.8G used of disk space
>> > and once we clean that goes down to 398M.
>> >
>> > The test case results are the obvious trick. I'm not sure if Jenkins
>> > keeps those somewhere else or if we need them to keep living in the
>> > target directories. If they need to stay in the target directories
>> > we can maybe create a script that deletes everything in target/ but
>> > surefire results.
>> >
>> > If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5
>> > gigabytes.
>> >
>> > Then we could just keep 30 builds regardless of days and it should be
>> > good.
>> >
>> > Thoughts?
>> >
>> >
>> > -David
>> >
>>
>
>
> --
> Atentamente:
> César Hernández.
>
--
Atentamente:
César Hernández.
Re: Increasing Discard Old Builds strategy in CI
Posted by Cesar Hernandez <ce...@gmail.com>.
In the job, after the test results are executed we can add a post-build
doing a:
rm -rf /home/jenkins/.m2/
El mar, 26 abr 2022 a las 12:51, Zowalla, Richard (<
richard.zowalla@hs-heilbronn.de>) escribió:
> Afaik Jenkins is parsing the surefire report files. The log files might
> also be important for the creator of the PR, so we would need to keep
> them too.
>
> Might be worth a try, I guess.
>
> Gruß
> Richard
>
> Am Dienstag, dem 26.04.2022 um 11:15 -0700 schrieb David Blevins:
> > > On Apr 26, 2022, at 10:47 AM, Zowalla, Richard <
> > > richard.zowalla@hs-heilbronn.de> wrote:
> > >
> > > However, we need to carefully monitor the disk usage / volume used
> > > as
> > > INFRA has a problem if our used storage volume per jobs exceeds a
> > > certain quota.
> > >
> > > We occupied 1,5T last week (cumulated over a few weeks) and they
> > > were a
> > > bit mad as the Jenkins CI env went out of storage.
> >
> > I was going to jump in on that thread and got side tracked.
> >
> > I wonder if we can do a `mvn clean` or something similar as a post
> > build step. Looks like after a build we have 2.8G used of disk space
> > and once we clean that goes down to 398M.
> >
> > The test case results are the obvious trick. I'm not sure if Jenkins
> > keeps those somewhere else or if we need them to keep living in the
> > target directories. If they need to stay in the target directories
> > we can maybe create a script that deletes everything in target/ but
> > surefire results.
> >
> > If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5
> > gigabytes.
> >
> > Then we could just keep 30 builds regardless of days and it should be
> > good.
> >
> > Thoughts?
> >
> >
> > -David
> >
>
--
Atentamente:
César Hernández.
Re: Increasing Discard Old Builds strategy in CI
Posted by "Zowalla, Richard" <ri...@hs-heilbronn.de>.
Afaik Jenkins is parsing the surefire report files. The log files might
also be important for the creator of the PR, so we would need to keep
them too.
Might be worth a try, I guess.
Gruß
Richard
Am Dienstag, dem 26.04.2022 um 11:15 -0700 schrieb David Blevins:
> > On Apr 26, 2022, at 10:47 AM, Zowalla, Richard <
> > richard.zowalla@hs-heilbronn.de> wrote:
> >
> > However, we need to carefully monitor the disk usage / volume used
> > as
> > INFRA has a problem if our used storage volume per jobs exceeds a
> > certain quota.
> >
> > We occupied 1,5T last week (cumulated over a few weeks) and they
> > were a
> > bit mad as the Jenkins CI env went out of storage.
>
> I was going to jump in on that thread and got side tracked.
>
> I wonder if we can do a `mvn clean` or something similar as a post
> build step. Looks like after a build we have 2.8G used of disk space
> and once we clean that goes down to 398M.
>
> The test case results are the obvious trick. I'm not sure if Jenkins
> keeps those somewhere else or if we need them to keep living in the
> target directories. If they need to stay in the target directories
> we can maybe create a script that deletes everything in target/ but
> surefire results.
>
> If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5
> gigabytes.
>
> Then we could just keep 30 builds regardless of days and it should be
> good.
>
> Thoughts?
>
>
> -David
>
Re: Increasing Discard Old Builds strategy in CI
Posted by David Blevins <da...@gmail.com>.
> On Apr 26, 2022, at 10:47 AM, Zowalla, Richard <ri...@hs-heilbronn.de> wrote:
>
> However, we need to carefully monitor the disk usage / volume used as
> INFRA has a problem if our used storage volume per jobs exceeds a
> certain quota.
>
> We occupied 1,5T last week (cumulated over a few weeks) and they were a
> bit mad as the Jenkins CI env went out of storage.
I was going to jump in on that thread and got side tracked.
I wonder if we can do a `mvn clean` or something similar as a post build step. Looks like after a build we have 2.8G used of disk space and once we clean that goes down to 398M.
The test case results are the obvious trick. I'm not sure if Jenkins keeps those somewhere else or if we need them to keep living in the target directories. If they need to stay in the target directories we can maybe create a script that deletes everything in target/ but surefire results.
If we did that, we'd use 1.5 gigabytes for 30 builds vs 10.5 gigabytes.
Then we could just keep 30 builds regardless of days and it should be good.
Thoughts?
-David
Re: Increasing Discard Old Builds strategy in CI
Posted by "Zowalla, Richard" <ri...@hs-heilbronn.de>.
Hi Cesar,
I have no problem with increasing the values for the discard strategy.
It makes sense to keep it a bit longer for PRs.
However, we need to carefully monitor the disk usage / volume used as
INFRA has a problem if our used storage volume per jobs exceeds a
certain quota.
We occupied 1,5T last week (cumulated over a few weeks) and they were a
bit mad as the Jenkins CI env went out of storage.
Gruß
Richard
Am Dienstag, dem 26.04.2022 um 11:39 -0600 schrieb Cesar Hernandez:
> Hello,
>
> Our current Discard old builds Strategy in CI for PR's for the master
> branch is [1]:
> Days to keep builds: 2
> Max # of builds to keep: 3
>
> Is there any objection if I update this to at least:
> Days to keep builds: 15
> Max # of builds to keep: 30
>
> Currently, if one opens a PR on a Friday and tries to catch up on
> mid-Monday next week, the PR results are gone.
> The above proposal will also give us a bit more room for more context
> on
> how PR's and past builds have evolved, especially when trying to
> chase and
> resolve random test failures.
>
>
> [1]
> https://ci-builds.apache.org/job/Tomee/job/master-pull-request