You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@solr.apache.org by Robert Thomson <bt...@apache.org> on 2026/06/29 07:04:23 UTC
ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Hi,
We are regularly noticing a build-up of large ws-cleanup directories
on lucene3 and 4, necessitating deletion. Camel also experiences this
issue with builds and the root cause is that there are files created
as part of the docker build that are not owned by the user used for
the cleanup, so fail to be deleted by the workspace clean up plugin.
Searching with GenAI came up with the following suggestion when
discussing with them that you may like to try out in your pipelines
that create docker images.
Kind regards,
-Bob Thomson,
ASF Infrastructure
post {
always {
// Fallback option: If standard cleanWs() stalls out on permissions
script {
try {
cleanWs deleteDirs: true, disableDeferredWipeout: true
} catch (Exception e) {
echo “Standard cleanup failed, launching Docker Root
Purge force-override...”
sh “”"
docker run --rm \
-v ${WORKSPACE}/..:/zap \
alpine:latest \
sh -c ‘rm -rf /zap/*-ws-cleanup*’
“”"
}
}
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by Bob Thomson <bt...@apache.org>.
The cleanup setting on the job only applies to that job config. I think it can be set more globally, but not looked into the details as yet.
Absolutely no ws-cleanup dirs on lucene3, so looks great!
On lucene4 there are a few:
solr-desktop-ui-nightly_ws-cleanup_1784131113250
Solr-TestIntegration-9.11_ws-cleanup_1784154127477
Solr-Lint-10.x_ws-cleanup_1784162700403
Solr-Artifacts-9.11_ws-cleanup_1784170570055
Solr-Test-9.11_ws-cleanup_1784113968538
Solr-Test-9.x_ws-cleanup_1784118593979
Solr-Test-10.x_ws-cleanup_1784121394984
Solr-Test-10.0_ws-cleanup_1784125677574
Solr-Test-10.0_ws-cleanup_1784149989136
Solr-Test-10.0_ws-cleanup_1784172720758
Solr-Test-10.0_ws-cleanup_1784182295683
Kind regards,
-Bob Thomson,
ASF Infrastructure
On 2026/07/16 01:23:48 David Smiley wrote:
> ok cool. I enabled this flag on several more jobs -- all the bulk test
> ones for the different Solr non-release branches. I assume that cleanup
> needn't be tied to any particular job so much as the cleanup needs to
> happen enough that it executes on the 2 build machines.
>
> On Wed, Jul 15, 2026 at 6:52 AM Bob Thomson <bt...@apache.org> wrote:
>
> > That one claims to have cleared the workspace and it didn't seem to affect
> > the job duration at all either which is good:
> >
> > [WS-CLEANUP] Deleting project workspace...
> > [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
> > [WS-CLEANUP] Done
> >
> > and that worked, as I don't see a ws-cleanup for Solr-Test-main after this
> > one:
> >
> > Jul 14 02:12 Solr-Test-main_ws-cleanup_1784004182558
> >
> > I'll remove all the other, olrder Solr-Test-main_ws-cleanup_1* dirs on
> > lucene3/4, and we can monitor and confirm no new ones get created/left. If
> > so, that seems a valid setting for all of the jobs that run on lucene3/4.
> > We can then see if it's possible to enable it at a higher level so all the
> > jobs run that way, provided you don't see any longer duration on job run
> > time due to this.
> >
> > Kind regards,
> > -Bob Thomson,
> > ASF Infrastructure
> >
> > On 2026/07/15 01:09:34 David Smiley wrote:
> > > I checked "Disable deferred wipeout" and scheduled a build. It ran a few
> > > hours ago on Lucene-3
> > > https://ci-builds.apache.org/job/Solr/job/Solr-Test-main/14288/
> > >
> > > On Tue, Jul 14, 2026 at 9:55 AM Bob Thomson <bt...@apache.org> wrote:
> > >
> > > > One thing you could try, to see if it helps, is Disable Deferred
> > Wipeout -
> > > > this will cause a delay in your job completing, but it means the
> > workspace
> > > > cleanup happens before the job carries on, rather than being offloaded
> > to a
> > > > thread to do later, something which seems to be not working properly
> > at the
> > > > moment.
> > > >
> > > > It's under:
> > > > Build Environment, Delete workspace before build starts, Advanced:
> > > >
> > > > Disable deferred wipeout
> > > >
> > > > On lucene 3, there are several ws-cleanup directories for
> > Solr-Test-main,
> > > > so that would seem a good job to try this out on. If it makes the job
> > fail,
> > > > we will hopefully get info on why, if not, we can see what sort of
> > delay is
> > > > incurred.
> > > >
> > > > The explanation from Gemin Gen AI:
> > > > "In a Freestyle job, when the "deferred wipeout" feature is active,
> > > > Jenkins instantly renames your workspace directory to
> > > > workspace_ws-cleanup_[timestamp] and tells the build that it's
> > finished. It
> > > > then delegates the actual deletion of that directory to a separate Java
> > > > thread in the background.
> > > >
> > > > By checking Disable deferred wipeout, you are telling Jenkins: "Do not
> > use
> > > > background threads. Hold up the build execution for an extra second,
> > delete
> > > > the files synchronously right now, and only mark the build complete
> > once
> > > > they are 100% gone.""
> > > >
> > > >
> > > > On 2026/07/14 12:49:48 David Smiley wrote:
> > > > > I modified one build,
> > > > > https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/
> > to
> > > > > have an additional shell execution step with that snippet, which
> > performs
> > > > > the cleanup via Docker as you suggested, albeit not with the
> > > > Jenkins/groovy
> > > > > wrapper all around it. It ran successfully this morning. I'm not
> > sure
> > > > why
> > > > > Docker is involved for what looks like a simple file removal; might
> > you
> > > > > explain?
> > > > >
> > > > > On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <bt...@apache.org>
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > In Infra, we don't work on job definitions as that is a project's
> > > > domain.
> > > > > > The above is only a suggestion which may or may not help.
> > > > > > If it is easier, you could look at adding similar code to the jobs
> > as a
> > > > > > shell script post build step in order to remove the files. Aurélien
> > > > Pupier
> > > > > > of the Camel project has been looking at the same issue there, so
> > worth
> > > > > > having a chat with them about potential solutions.
> > > > > >
> > > > > > In terms of the jobs which are leaving ws-cleanup directories
> > behind,
> > > > it
> > > > > > seems to be many of them. On lucene 3 today, there are already 113
> > such
> > > > > > directories, 213GB so far.
> > > > > >
> > > > > > the jobs that they are associated with, from per their directory
> > names
> > > > are:
> > > > > >
> > > > > > OWASP-9.x
> > > > > > OWASP-main
> > > > > > Solr-Artifacts-10.x
> > > > > > Solr-Artifacts-9.11
> > > > > > Solr-Artifacts-9.x
> > > > > > Solr-Artifacts-main
> > > > > > Solr-BadApples-Tests-main
> > > > > > Solr-Docker-Nightly-9.11
> > > > > > Solr-Lint-10.x
> > > > > > Solr-Lint-9.11
> > > > > > Solr-Lint-9.x
> > > > > > Solr-Lint-main
> > > > > > Solr-Maven-Snapshots-main
> > > > > > Solr-Test-10.0
> > > > > > Solr-Test-10.x
> > > > > > Solr-Test-9.11
> > > > > > Solr-Test-9.x
> > > > > > Solr-Test-main
> > > > > > Solr-TestIntegration-10.x
> > > > > > Solr-TestIntegration-9.11
> > > > > > Solr-TestIntegration-9.x
> > > > > > Solr-TestIntegration-main
> > > > > > solr-desktop-ui-nightly
> > > > > >
> > > > > > On 4, there are 80 from these builds:
> > > > > >
> > > > > > OWASP-9.x
> > > > > > OWASP-main
> > > > > > Solr-Artifacts-10.x
> > > > > > Solr-Artifacts-9.11
> > > > > > Solr-Artifacts-9.x
> > > > > > Solr-Docker-Nightly-9.11
> > > > > > Solr-Lint-10.x
> > > > > > Solr-Lint-9.11
> > > > > > Solr-Lint-9.x
> > > > > > Solr-Lint-main
> > > > > > Solr-Maven-Snapshots-main
> > > > > > Solr-Test-10.0
> > > > > > Solr-Test-10.x
> > > > > > Solr-Test-9.11
> > > > > > Solr-Test-9.x
> > > > > > Solr-Test-main
> > > > > > Solr-TestIntegration-9.11
> > > > > > Solr-TestIntegration-9.x
> > > > > >
> > > > > > On 2026/07/03 20:32:09 David Smiley wrote:
> > > > > > > I know too little about Jenkins to know how to apply this
> > solution,
> > > > but I
> > > > > > > can try to help. The Solr jobs are defined in Jenkins, not in
> > our
> > > > repo
> > > > > > in
> > > > > > > Jenkinsfile. Maybe you could apply this to one of the jobs
> > yourself?
> > > > > > Does
> > > > > > > this need to be done on all jobs or just some or will just one
> > do?
> > > > > > >
> > > > > > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org>
> > > > wrote:
> > > > > > >
> > > > > > > > This has come up as an issue again today, necessitating
> > > > intervention to
> > > > > > > > keep the service available.
> > > > > > > >
> > > > > > > > On 2026/06/29 07:04:23 Robert Thomson wrote:
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > We are regularly noticing a build-up of large ws-cleanup
> > > > directories
> > > > > > > > > on lucene3 and 4, necessitating deletion. Camel also
> > experiences
> > > > this
> > > > > > > > > issue with builds and the root cause is that there are files
> > > > created
> > > > > > > > > as part of the docker build that are not owned by the user
> > used
> > > > for
> > > > > > > > > the cleanup, so fail to be deleted by the workspace clean up
> > > > plugin.
> > > > > > > > >
> > > > > > > > > Searching with GenAI came up with the following suggestion
> > when
> > > > > > > > > discussing with them that you may like to try out in your
> > > > pipelines
> > > > > > > > > that create docker images.
> > > > > > > > >
> > > > > > > > > Kind regards,
> > > > > > > > > -Bob Thomson,
> > > > > > > > > ASF Infrastructure
> > > > > > > > >
> > > > > > > > > post {
> > > > > > > > > always {
> > > > > > > > > // Fallback option: If standard cleanWs() stalls out
> > on
> > > > > > > > permissions
> > > > > > > > > script {
> > > > > > > > > try {
> > > > > > > > > cleanWs deleteDirs: true,
> > disableDeferredWipeout:
> > > > > > true
> > > > > > > > > } catch (Exception e) {
> > > > > > > > > echo “Standard cleanup failed, launching
> > Docker
> > > > Root
> > > > > > > > > Purge force-override...”
> > > > > > > > > sh “”"
> > > > > > > > > docker run --rm \
> > > > > > > > > -v ${WORKSPACE}/..:/zap \
> > > > > > > > > alpine:latest \
> > > > > > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > > > > > > > > “”"
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > > }
> > > > > > > > >
> > > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by David Smiley <ds...@apache.org>.
ok cool. I enabled this flag on several more jobs -- all the bulk test
ones for the different Solr non-release branches. I assume that cleanup
needn't be tied to any particular job so much as the cleanup needs to
happen enough that it executes on the 2 build machines.
On Wed, Jul 15, 2026 at 6:52 AM Bob Thomson <bt...@apache.org> wrote:
> That one claims to have cleared the workspace and it didn't seem to affect
> the job duration at all either which is good:
>
> [WS-CLEANUP] Deleting project workspace...
> [WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
> [WS-CLEANUP] Done
>
> and that worked, as I don't see a ws-cleanup for Solr-Test-main after this
> one:
>
> Jul 14 02:12 Solr-Test-main_ws-cleanup_1784004182558
>
> I'll remove all the other, olrder Solr-Test-main_ws-cleanup_1* dirs on
> lucene3/4, and we can monitor and confirm no new ones get created/left. If
> so, that seems a valid setting for all of the jobs that run on lucene3/4.
> We can then see if it's possible to enable it at a higher level so all the
> jobs run that way, provided you don't see any longer duration on job run
> time due to this.
>
> Kind regards,
> -Bob Thomson,
> ASF Infrastructure
>
> On 2026/07/15 01:09:34 David Smiley wrote:
> > I checked "Disable deferred wipeout" and scheduled a build. It ran a few
> > hours ago on Lucene-3
> > https://ci-builds.apache.org/job/Solr/job/Solr-Test-main/14288/
> >
> > On Tue, Jul 14, 2026 at 9:55 AM Bob Thomson <bt...@apache.org> wrote:
> >
> > > One thing you could try, to see if it helps, is Disable Deferred
> Wipeout -
> > > this will cause a delay in your job completing, but it means the
> workspace
> > > cleanup happens before the job carries on, rather than being offloaded
> to a
> > > thread to do later, something which seems to be not working properly
> at the
> > > moment.
> > >
> > > It's under:
> > > Build Environment, Delete workspace before build starts, Advanced:
> > >
> > > Disable deferred wipeout
> > >
> > > On lucene 3, there are several ws-cleanup directories for
> Solr-Test-main,
> > > so that would seem a good job to try this out on. If it makes the job
> fail,
> > > we will hopefully get info on why, if not, we can see what sort of
> delay is
> > > incurred.
> > >
> > > The explanation from Gemin Gen AI:
> > > "In a Freestyle job, when the "deferred wipeout" feature is active,
> > > Jenkins instantly renames your workspace directory to
> > > workspace_ws-cleanup_[timestamp] and tells the build that it's
> finished. It
> > > then delegates the actual deletion of that directory to a separate Java
> > > thread in the background.
> > >
> > > By checking Disable deferred wipeout, you are telling Jenkins: "Do not
> use
> > > background threads. Hold up the build execution for an extra second,
> delete
> > > the files synchronously right now, and only mark the build complete
> once
> > > they are 100% gone.""
> > >
> > >
> > > On 2026/07/14 12:49:48 David Smiley wrote:
> > > > I modified one build,
> > > > https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/
> to
> > > > have an additional shell execution step with that snippet, which
> performs
> > > > the cleanup via Docker as you suggested, albeit not with the
> > > Jenkins/groovy
> > > > wrapper all around it. It ran successfully this morning. I'm not
> sure
> > > why
> > > > Docker is involved for what looks like a simple file removal; might
> you
> > > > explain?
> > > >
> > > > On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <bt...@apache.org>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > In Infra, we don't work on job definitions as that is a project's
> > > domain.
> > > > > The above is only a suggestion which may or may not help.
> > > > > If it is easier, you could look at adding similar code to the jobs
> as a
> > > > > shell script post build step in order to remove the files. Aurélien
> > > Pupier
> > > > > of the Camel project has been looking at the same issue there, so
> worth
> > > > > having a chat with them about potential solutions.
> > > > >
> > > > > In terms of the jobs which are leaving ws-cleanup directories
> behind,
> > > it
> > > > > seems to be many of them. On lucene 3 today, there are already 113
> such
> > > > > directories, 213GB so far.
> > > > >
> > > > > the jobs that they are associated with, from per their directory
> names
> > > are:
> > > > >
> > > > > OWASP-9.x
> > > > > OWASP-main
> > > > > Solr-Artifacts-10.x
> > > > > Solr-Artifacts-9.11
> > > > > Solr-Artifacts-9.x
> > > > > Solr-Artifacts-main
> > > > > Solr-BadApples-Tests-main
> > > > > Solr-Docker-Nightly-9.11
> > > > > Solr-Lint-10.x
> > > > > Solr-Lint-9.11
> > > > > Solr-Lint-9.x
> > > > > Solr-Lint-main
> > > > > Solr-Maven-Snapshots-main
> > > > > Solr-Test-10.0
> > > > > Solr-Test-10.x
> > > > > Solr-Test-9.11
> > > > > Solr-Test-9.x
> > > > > Solr-Test-main
> > > > > Solr-TestIntegration-10.x
> > > > > Solr-TestIntegration-9.11
> > > > > Solr-TestIntegration-9.x
> > > > > Solr-TestIntegration-main
> > > > > solr-desktop-ui-nightly
> > > > >
> > > > > On 4, there are 80 from these builds:
> > > > >
> > > > > OWASP-9.x
> > > > > OWASP-main
> > > > > Solr-Artifacts-10.x
> > > > > Solr-Artifacts-9.11
> > > > > Solr-Artifacts-9.x
> > > > > Solr-Docker-Nightly-9.11
> > > > > Solr-Lint-10.x
> > > > > Solr-Lint-9.11
> > > > > Solr-Lint-9.x
> > > > > Solr-Lint-main
> > > > > Solr-Maven-Snapshots-main
> > > > > Solr-Test-10.0
> > > > > Solr-Test-10.x
> > > > > Solr-Test-9.11
> > > > > Solr-Test-9.x
> > > > > Solr-Test-main
> > > > > Solr-TestIntegration-9.11
> > > > > Solr-TestIntegration-9.x
> > > > >
> > > > > On 2026/07/03 20:32:09 David Smiley wrote:
> > > > > > I know too little about Jenkins to know how to apply this
> solution,
> > > but I
> > > > > > can try to help. The Solr jobs are defined in Jenkins, not in
> our
> > > repo
> > > > > in
> > > > > > Jenkinsfile. Maybe you could apply this to one of the jobs
> yourself?
> > > > > Does
> > > > > > this need to be done on all jobs or just some or will just one
> do?
> > > > > >
> > > > > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org>
> > > wrote:
> > > > > >
> > > > > > > This has come up as an issue again today, necessitating
> > > intervention to
> > > > > > > keep the service available.
> > > > > > >
> > > > > > > On 2026/06/29 07:04:23 Robert Thomson wrote:
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > We are regularly noticing a build-up of large ws-cleanup
> > > directories
> > > > > > > > on lucene3 and 4, necessitating deletion. Camel also
> experiences
> > > this
> > > > > > > > issue with builds and the root cause is that there are files
> > > created
> > > > > > > > as part of the docker build that are not owned by the user
> used
> > > for
> > > > > > > > the cleanup, so fail to be deleted by the workspace clean up
> > > plugin.
> > > > > > > >
> > > > > > > > Searching with GenAI came up with the following suggestion
> when
> > > > > > > > discussing with them that you may like to try out in your
> > > pipelines
> > > > > > > > that create docker images.
> > > > > > > >
> > > > > > > > Kind regards,
> > > > > > > > -Bob Thomson,
> > > > > > > > ASF Infrastructure
> > > > > > > >
> > > > > > > > post {
> > > > > > > > always {
> > > > > > > > // Fallback option: If standard cleanWs() stalls out
> on
> > > > > > > permissions
> > > > > > > > script {
> > > > > > > > try {
> > > > > > > > cleanWs deleteDirs: true,
> disableDeferredWipeout:
> > > > > true
> > > > > > > > } catch (Exception e) {
> > > > > > > > echo “Standard cleanup failed, launching
> Docker
> > > Root
> > > > > > > > Purge force-override...”
> > > > > > > > sh “”"
> > > > > > > > docker run --rm \
> > > > > > > > -v ${WORKSPACE}/..:/zap \
> > > > > > > > alpine:latest \
> > > > > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > > > > > > > “”"
> > > > > > > > }
> > > > > > > > }
> > > > > > > > }
> > > > > > > > }
> > > > > > > >
> > > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by Bob Thomson <bt...@apache.org>.
That one claims to have cleared the workspace and it didn't seem to affect the job duration at all either which is good:
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is disabled by the job configuration...
[WS-CLEANUP] Done
and that worked, as I don't see a ws-cleanup for Solr-Test-main after this one:
Jul 14 02:12 Solr-Test-main_ws-cleanup_1784004182558
I'll remove all the other, olrder Solr-Test-main_ws-cleanup_1* dirs on lucene3/4, and we can monitor and confirm no new ones get created/left. If so, that seems a valid setting for all of the jobs that run on lucene3/4. We can then see if it's possible to enable it at a higher level so all the jobs run that way, provided you don't see any longer duration on job run time due to this.
Kind regards,
-Bob Thomson,
ASF Infrastructure
On 2026/07/15 01:09:34 David Smiley wrote:
> I checked "Disable deferred wipeout" and scheduled a build. It ran a few
> hours ago on Lucene-3
> https://ci-builds.apache.org/job/Solr/job/Solr-Test-main/14288/
>
> On Tue, Jul 14, 2026 at 9:55 AM Bob Thomson <bt...@apache.org> wrote:
>
> > One thing you could try, to see if it helps, is Disable Deferred Wipeout -
> > this will cause a delay in your job completing, but it means the workspace
> > cleanup happens before the job carries on, rather than being offloaded to a
> > thread to do later, something which seems to be not working properly at the
> > moment.
> >
> > It's under:
> > Build Environment, Delete workspace before build starts, Advanced:
> >
> > Disable deferred wipeout
> >
> > On lucene 3, there are several ws-cleanup directories for Solr-Test-main,
> > so that would seem a good job to try this out on. If it makes the job fail,
> > we will hopefully get info on why, if not, we can see what sort of delay is
> > incurred.
> >
> > The explanation from Gemin Gen AI:
> > "In a Freestyle job, when the "deferred wipeout" feature is active,
> > Jenkins instantly renames your workspace directory to
> > workspace_ws-cleanup_[timestamp] and tells the build that it's finished. It
> > then delegates the actual deletion of that directory to a separate Java
> > thread in the background.
> >
> > By checking Disable deferred wipeout, you are telling Jenkins: "Do not use
> > background threads. Hold up the build execution for an extra second, delete
> > the files synchronously right now, and only mark the build complete once
> > they are 100% gone.""
> >
> >
> > On 2026/07/14 12:49:48 David Smiley wrote:
> > > I modified one build,
> > > https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ to
> > > have an additional shell execution step with that snippet, which performs
> > > the cleanup via Docker as you suggested, albeit not with the
> > Jenkins/groovy
> > > wrapper all around it. It ran successfully this morning. I'm not sure
> > why
> > > Docker is involved for what looks like a simple file removal; might you
> > > explain?
> > >
> > > On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <bt...@apache.org> wrote:
> > >
> > > > Hi,
> > > >
> > > > In Infra, we don't work on job definitions as that is a project's
> > domain.
> > > > The above is only a suggestion which may or may not help.
> > > > If it is easier, you could look at adding similar code to the jobs as a
> > > > shell script post build step in order to remove the files. Aurélien
> > Pupier
> > > > of the Camel project has been looking at the same issue there, so worth
> > > > having a chat with them about potential solutions.
> > > >
> > > > In terms of the jobs which are leaving ws-cleanup directories behind,
> > it
> > > > seems to be many of them. On lucene 3 today, there are already 113 such
> > > > directories, 213GB so far.
> > > >
> > > > the jobs that they are associated with, from per their directory names
> > are:
> > > >
> > > > OWASP-9.x
> > > > OWASP-main
> > > > Solr-Artifacts-10.x
> > > > Solr-Artifacts-9.11
> > > > Solr-Artifacts-9.x
> > > > Solr-Artifacts-main
> > > > Solr-BadApples-Tests-main
> > > > Solr-Docker-Nightly-9.11
> > > > Solr-Lint-10.x
> > > > Solr-Lint-9.11
> > > > Solr-Lint-9.x
> > > > Solr-Lint-main
> > > > Solr-Maven-Snapshots-main
> > > > Solr-Test-10.0
> > > > Solr-Test-10.x
> > > > Solr-Test-9.11
> > > > Solr-Test-9.x
> > > > Solr-Test-main
> > > > Solr-TestIntegration-10.x
> > > > Solr-TestIntegration-9.11
> > > > Solr-TestIntegration-9.x
> > > > Solr-TestIntegration-main
> > > > solr-desktop-ui-nightly
> > > >
> > > > On 4, there are 80 from these builds:
> > > >
> > > > OWASP-9.x
> > > > OWASP-main
> > > > Solr-Artifacts-10.x
> > > > Solr-Artifacts-9.11
> > > > Solr-Artifacts-9.x
> > > > Solr-Docker-Nightly-9.11
> > > > Solr-Lint-10.x
> > > > Solr-Lint-9.11
> > > > Solr-Lint-9.x
> > > > Solr-Lint-main
> > > > Solr-Maven-Snapshots-main
> > > > Solr-Test-10.0
> > > > Solr-Test-10.x
> > > > Solr-Test-9.11
> > > > Solr-Test-9.x
> > > > Solr-Test-main
> > > > Solr-TestIntegration-9.11
> > > > Solr-TestIntegration-9.x
> > > >
> > > > On 2026/07/03 20:32:09 David Smiley wrote:
> > > > > I know too little about Jenkins to know how to apply this solution,
> > but I
> > > > > can try to help. The Solr jobs are defined in Jenkins, not in our
> > repo
> > > > in
> > > > > Jenkinsfile. Maybe you could apply this to one of the jobs yourself?
> > > > Does
> > > > > this need to be done on all jobs or just some or will just one do?
> > > > >
> > > > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org>
> > wrote:
> > > > >
> > > > > > This has come up as an issue again today, necessitating
> > intervention to
> > > > > > keep the service available.
> > > > > >
> > > > > > On 2026/06/29 07:04:23 Robert Thomson wrote:
> > > > > > > Hi,
> > > > > > >
> > > > > > > We are regularly noticing a build-up of large ws-cleanup
> > directories
> > > > > > > on lucene3 and 4, necessitating deletion. Camel also experiences
> > this
> > > > > > > issue with builds and the root cause is that there are files
> > created
> > > > > > > as part of the docker build that are not owned by the user used
> > for
> > > > > > > the cleanup, so fail to be deleted by the workspace clean up
> > plugin.
> > > > > > >
> > > > > > > Searching with GenAI came up with the following suggestion when
> > > > > > > discussing with them that you may like to try out in your
> > pipelines
> > > > > > > that create docker images.
> > > > > > >
> > > > > > > Kind regards,
> > > > > > > -Bob Thomson,
> > > > > > > ASF Infrastructure
> > > > > > >
> > > > > > > post {
> > > > > > > always {
> > > > > > > // Fallback option: If standard cleanWs() stalls out on
> > > > > > permissions
> > > > > > > script {
> > > > > > > try {
> > > > > > > cleanWs deleteDirs: true, disableDeferredWipeout:
> > > > true
> > > > > > > } catch (Exception e) {
> > > > > > > echo “Standard cleanup failed, launching Docker
> > Root
> > > > > > > Purge force-override...”
> > > > > > > sh “”"
> > > > > > > docker run --rm \
> > > > > > > -v ${WORKSPACE}/..:/zap \
> > > > > > > alpine:latest \
> > > > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > > > > > > “”"
> > > > > > > }
> > > > > > > }
> > > > > > > }
> > > > > > > }
> > > > > > >
> > > > > > >
> > ---------------------------------------------------------------------
> > > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > >
> > > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by David Smiley <ds...@apache.org>.
I checked "Disable deferred wipeout" and scheduled a build. It ran a few
hours ago on Lucene-3
https://ci-builds.apache.org/job/Solr/job/Solr-Test-main/14288/
On Tue, Jul 14, 2026 at 9:55 AM Bob Thomson <bt...@apache.org> wrote:
> One thing you could try, to see if it helps, is Disable Deferred Wipeout -
> this will cause a delay in your job completing, but it means the workspace
> cleanup happens before the job carries on, rather than being offloaded to a
> thread to do later, something which seems to be not working properly at the
> moment.
>
> It's under:
> Build Environment, Delete workspace before build starts, Advanced:
>
> Disable deferred wipeout
>
> On lucene 3, there are several ws-cleanup directories for Solr-Test-main,
> so that would seem a good job to try this out on. If it makes the job fail,
> we will hopefully get info on why, if not, we can see what sort of delay is
> incurred.
>
> The explanation from Gemin Gen AI:
> "In a Freestyle job, when the "deferred wipeout" feature is active,
> Jenkins instantly renames your workspace directory to
> workspace_ws-cleanup_[timestamp] and tells the build that it's finished. It
> then delegates the actual deletion of that directory to a separate Java
> thread in the background.
>
> By checking Disable deferred wipeout, you are telling Jenkins: "Do not use
> background threads. Hold up the build execution for an extra second, delete
> the files synchronously right now, and only mark the build complete once
> they are 100% gone.""
>
>
> On 2026/07/14 12:49:48 David Smiley wrote:
> > I modified one build,
> > https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ to
> > have an additional shell execution step with that snippet, which performs
> > the cleanup via Docker as you suggested, albeit not with the
> Jenkins/groovy
> > wrapper all around it. It ran successfully this morning. I'm not sure
> why
> > Docker is involved for what looks like a simple file removal; might you
> > explain?
> >
> > On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <bt...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > In Infra, we don't work on job definitions as that is a project's
> domain.
> > > The above is only a suggestion which may or may not help.
> > > If it is easier, you could look at adding similar code to the jobs as a
> > > shell script post build step in order to remove the files. Aurélien
> Pupier
> > > of the Camel project has been looking at the same issue there, so worth
> > > having a chat with them about potential solutions.
> > >
> > > In terms of the jobs which are leaving ws-cleanup directories behind,
> it
> > > seems to be many of them. On lucene 3 today, there are already 113 such
> > > directories, 213GB so far.
> > >
> > > the jobs that they are associated with, from per their directory names
> are:
> > >
> > > OWASP-9.x
> > > OWASP-main
> > > Solr-Artifacts-10.x
> > > Solr-Artifacts-9.11
> > > Solr-Artifacts-9.x
> > > Solr-Artifacts-main
> > > Solr-BadApples-Tests-main
> > > Solr-Docker-Nightly-9.11
> > > Solr-Lint-10.x
> > > Solr-Lint-9.11
> > > Solr-Lint-9.x
> > > Solr-Lint-main
> > > Solr-Maven-Snapshots-main
> > > Solr-Test-10.0
> > > Solr-Test-10.x
> > > Solr-Test-9.11
> > > Solr-Test-9.x
> > > Solr-Test-main
> > > Solr-TestIntegration-10.x
> > > Solr-TestIntegration-9.11
> > > Solr-TestIntegration-9.x
> > > Solr-TestIntegration-main
> > > solr-desktop-ui-nightly
> > >
> > > On 4, there are 80 from these builds:
> > >
> > > OWASP-9.x
> > > OWASP-main
> > > Solr-Artifacts-10.x
> > > Solr-Artifacts-9.11
> > > Solr-Artifacts-9.x
> > > Solr-Docker-Nightly-9.11
> > > Solr-Lint-10.x
> > > Solr-Lint-9.11
> > > Solr-Lint-9.x
> > > Solr-Lint-main
> > > Solr-Maven-Snapshots-main
> > > Solr-Test-10.0
> > > Solr-Test-10.x
> > > Solr-Test-9.11
> > > Solr-Test-9.x
> > > Solr-Test-main
> > > Solr-TestIntegration-9.11
> > > Solr-TestIntegration-9.x
> > >
> > > On 2026/07/03 20:32:09 David Smiley wrote:
> > > > I know too little about Jenkins to know how to apply this solution,
> but I
> > > > can try to help. The Solr jobs are defined in Jenkins, not in our
> repo
> > > in
> > > > Jenkinsfile. Maybe you could apply this to one of the jobs yourself?
> > > Does
> > > > this need to be done on all jobs or just some or will just one do?
> > > >
> > > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org>
> wrote:
> > > >
> > > > > This has come up as an issue again today, necessitating
> intervention to
> > > > > keep the service available.
> > > > >
> > > > > On 2026/06/29 07:04:23 Robert Thomson wrote:
> > > > > > Hi,
> > > > > >
> > > > > > We are regularly noticing a build-up of large ws-cleanup
> directories
> > > > > > on lucene3 and 4, necessitating deletion. Camel also experiences
> this
> > > > > > issue with builds and the root cause is that there are files
> created
> > > > > > as part of the docker build that are not owned by the user used
> for
> > > > > > the cleanup, so fail to be deleted by the workspace clean up
> plugin.
> > > > > >
> > > > > > Searching with GenAI came up with the following suggestion when
> > > > > > discussing with them that you may like to try out in your
> pipelines
> > > > > > that create docker images.
> > > > > >
> > > > > > Kind regards,
> > > > > > -Bob Thomson,
> > > > > > ASF Infrastructure
> > > > > >
> > > > > > post {
> > > > > > always {
> > > > > > // Fallback option: If standard cleanWs() stalls out on
> > > > > permissions
> > > > > > script {
> > > > > > try {
> > > > > > cleanWs deleteDirs: true, disableDeferredWipeout:
> > > true
> > > > > > } catch (Exception e) {
> > > > > > echo “Standard cleanup failed, launching Docker
> Root
> > > > > > Purge force-override...”
> > > > > > sh “”"
> > > > > > docker run --rm \
> > > > > > -v ${WORKSPACE}/..:/zap \
> > > > > > alpine:latest \
> > > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > > > > > “”"
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > >
> > > > > >
> ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > >
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by Bob Thomson <bt...@apache.org>.
One thing you could try, to see if it helps, is Disable Deferred Wipeout - this will cause a delay in your job completing, but it means the workspace cleanup happens before the job carries on, rather than being offloaded to a thread to do later, something which seems to be not working properly at the moment.
It's under:
Build Environment, Delete workspace before build starts, Advanced:
Disable deferred wipeout
On lucene 3, there are several ws-cleanup directories for Solr-Test-main, so that would seem a good job to try this out on. If it makes the job fail, we will hopefully get info on why, if not, we can see what sort of delay is incurred.
The explanation from Gemin Gen AI:
"In a Freestyle job, when the "deferred wipeout" feature is active, Jenkins instantly renames your workspace directory to workspace_ws-cleanup_[timestamp] and tells the build that it's finished. It then delegates the actual deletion of that directory to a separate Java thread in the background.
By checking Disable deferred wipeout, you are telling Jenkins: "Do not use background threads. Hold up the build execution for an extra second, delete the files synchronously right now, and only mark the build complete once they are 100% gone.""
On 2026/07/14 12:49:48 David Smiley wrote:
> I modified one build,
> https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ to
> have an additional shell execution step with that snippet, which performs
> the cleanup via Docker as you suggested, albeit not with the Jenkins/groovy
> wrapper all around it. It ran successfully this morning. I'm not sure why
> Docker is involved for what looks like a simple file removal; might you
> explain?
>
> On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <bt...@apache.org> wrote:
>
> > Hi,
> >
> > In Infra, we don't work on job definitions as that is a project's domain.
> > The above is only a suggestion which may or may not help.
> > If it is easier, you could look at adding similar code to the jobs as a
> > shell script post build step in order to remove the files. Aurélien Pupier
> > of the Camel project has been looking at the same issue there, so worth
> > having a chat with them about potential solutions.
> >
> > In terms of the jobs which are leaving ws-cleanup directories behind, it
> > seems to be many of them. On lucene 3 today, there are already 113 such
> > directories, 213GB so far.
> >
> > the jobs that they are associated with, from per their directory names are:
> >
> > OWASP-9.x
> > OWASP-main
> > Solr-Artifacts-10.x
> > Solr-Artifacts-9.11
> > Solr-Artifacts-9.x
> > Solr-Artifacts-main
> > Solr-BadApples-Tests-main
> > Solr-Docker-Nightly-9.11
> > Solr-Lint-10.x
> > Solr-Lint-9.11
> > Solr-Lint-9.x
> > Solr-Lint-main
> > Solr-Maven-Snapshots-main
> > Solr-Test-10.0
> > Solr-Test-10.x
> > Solr-Test-9.11
> > Solr-Test-9.x
> > Solr-Test-main
> > Solr-TestIntegration-10.x
> > Solr-TestIntegration-9.11
> > Solr-TestIntegration-9.x
> > Solr-TestIntegration-main
> > solr-desktop-ui-nightly
> >
> > On 4, there are 80 from these builds:
> >
> > OWASP-9.x
> > OWASP-main
> > Solr-Artifacts-10.x
> > Solr-Artifacts-9.11
> > Solr-Artifacts-9.x
> > Solr-Docker-Nightly-9.11
> > Solr-Lint-10.x
> > Solr-Lint-9.11
> > Solr-Lint-9.x
> > Solr-Lint-main
> > Solr-Maven-Snapshots-main
> > Solr-Test-10.0
> > Solr-Test-10.x
> > Solr-Test-9.11
> > Solr-Test-9.x
> > Solr-Test-main
> > Solr-TestIntegration-9.11
> > Solr-TestIntegration-9.x
> >
> > On 2026/07/03 20:32:09 David Smiley wrote:
> > > I know too little about Jenkins to know how to apply this solution, but I
> > > can try to help. The Solr jobs are defined in Jenkins, not in our repo
> > in
> > > Jenkinsfile. Maybe you could apply this to one of the jobs yourself?
> > Does
> > > this need to be done on all jobs or just some or will just one do?
> > >
> > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org> wrote:
> > >
> > > > This has come up as an issue again today, necessitating intervention to
> > > > keep the service available.
> > > >
> > > > On 2026/06/29 07:04:23 Robert Thomson wrote:
> > > > > Hi,
> > > > >
> > > > > We are regularly noticing a build-up of large ws-cleanup directories
> > > > > on lucene3 and 4, necessitating deletion. Camel also experiences this
> > > > > issue with builds and the root cause is that there are files created
> > > > > as part of the docker build that are not owned by the user used for
> > > > > the cleanup, so fail to be deleted by the workspace clean up plugin.
> > > > >
> > > > > Searching with GenAI came up with the following suggestion when
> > > > > discussing with them that you may like to try out in your pipelines
> > > > > that create docker images.
> > > > >
> > > > > Kind regards,
> > > > > -Bob Thomson,
> > > > > ASF Infrastructure
> > > > >
> > > > > post {
> > > > > always {
> > > > > // Fallback option: If standard cleanWs() stalls out on
> > > > permissions
> > > > > script {
> > > > > try {
> > > > > cleanWs deleteDirs: true, disableDeferredWipeout:
> > true
> > > > > } catch (Exception e) {
> > > > > echo “Standard cleanup failed, launching Docker Root
> > > > > Purge force-override...”
> > > > > sh “”"
> > > > > docker run --rm \
> > > > > -v ${WORKSPACE}/..:/zap \
> > > > > alpine:latest \
> > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > > > > “”"
> > > > > }
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by Bob Thomson <bt...@apache.org>.
Having looked through the ws-cleanup dirs currently on lucene3 and 4, it does not look to be a permissions issue, as those dirs and all files in them are owned by jenkins, so it's a different issue in this case - further investigation needed.
On 2026/07/14 12:56:59 Bob Thomson wrote:
> Like I say, not my code as such, a Gen AI suggestion, but I would say it's an easy way for the process running as the jenkins user, to remove files which are not owned by the jenkins user, which seems to often be the reason for these ws-cleanup directories not being removed (Often in the Camel project case anyway). The other reason might be a restart of the server for example, which doesn't happen on lucene but sometimes does happen on some other CI servers we have due to Jenkins locking up (GC/RAM usage often).
>
> On 2026/07/14 12:49:48 David Smiley wrote:
> > I modified one build,
> > https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ to
> > have an additional shell execution step with that snippet, which performs
> > the cleanup via Docker as you suggested, albeit not with the Jenkins/groovy
> > wrapper all around it. It ran successfully this morning. I'm not sure why
> > Docker is involved for what looks like a simple file removal; might you
> > explain?
> >
> > On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <bt...@apache.org> wrote:
> >
> > > Hi,
> > >
> > > In Infra, we don't work on job definitions as that is a project's domain.
> > > The above is only a suggestion which may or may not help.
> > > If it is easier, you could look at adding similar code to the jobs as a
> > > shell script post build step in order to remove the files. Aurélien Pupier
> > > of the Camel project has been looking at the same issue there, so worth
> > > having a chat with them about potential solutions.
> > >
> > > In terms of the jobs which are leaving ws-cleanup directories behind, it
> > > seems to be many of them. On lucene 3 today, there are already 113 such
> > > directories, 213GB so far.
> > >
> > > the jobs that they are associated with, from per their directory names are:
> > >
> > > OWASP-9.x
> > > OWASP-main
> > > Solr-Artifacts-10.x
> > > Solr-Artifacts-9.11
> > > Solr-Artifacts-9.x
> > > Solr-Artifacts-main
> > > Solr-BadApples-Tests-main
> > > Solr-Docker-Nightly-9.11
> > > Solr-Lint-10.x
> > > Solr-Lint-9.11
> > > Solr-Lint-9.x
> > > Solr-Lint-main
> > > Solr-Maven-Snapshots-main
> > > Solr-Test-10.0
> > > Solr-Test-10.x
> > > Solr-Test-9.11
> > > Solr-Test-9.x
> > > Solr-Test-main
> > > Solr-TestIntegration-10.x
> > > Solr-TestIntegration-9.11
> > > Solr-TestIntegration-9.x
> > > Solr-TestIntegration-main
> > > solr-desktop-ui-nightly
> > >
> > > On 4, there are 80 from these builds:
> > >
> > > OWASP-9.x
> > > OWASP-main
> > > Solr-Artifacts-10.x
> > > Solr-Artifacts-9.11
> > > Solr-Artifacts-9.x
> > > Solr-Docker-Nightly-9.11
> > > Solr-Lint-10.x
> > > Solr-Lint-9.11
> > > Solr-Lint-9.x
> > > Solr-Lint-main
> > > Solr-Maven-Snapshots-main
> > > Solr-Test-10.0
> > > Solr-Test-10.x
> > > Solr-Test-9.11
> > > Solr-Test-9.x
> > > Solr-Test-main
> > > Solr-TestIntegration-9.11
> > > Solr-TestIntegration-9.x
> > >
> > > On 2026/07/03 20:32:09 David Smiley wrote:
> > > > I know too little about Jenkins to know how to apply this solution, but I
> > > > can try to help. The Solr jobs are defined in Jenkins, not in our repo
> > > in
> > > > Jenkinsfile. Maybe you could apply this to one of the jobs yourself?
> > > Does
> > > > this need to be done on all jobs or just some or will just one do?
> > > >
> > > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org> wrote:
> > > >
> > > > > This has come up as an issue again today, necessitating intervention to
> > > > > keep the service available.
> > > > >
> > > > > On 2026/06/29 07:04:23 Robert Thomson wrote:
> > > > > > Hi,
> > > > > >
> > > > > > We are regularly noticing a build-up of large ws-cleanup directories
> > > > > > on lucene3 and 4, necessitating deletion. Camel also experiences this
> > > > > > issue with builds and the root cause is that there are files created
> > > > > > as part of the docker build that are not owned by the user used for
> > > > > > the cleanup, so fail to be deleted by the workspace clean up plugin.
> > > > > >
> > > > > > Searching with GenAI came up with the following suggestion when
> > > > > > discussing with them that you may like to try out in your pipelines
> > > > > > that create docker images.
> > > > > >
> > > > > > Kind regards,
> > > > > > -Bob Thomson,
> > > > > > ASF Infrastructure
> > > > > >
> > > > > > post {
> > > > > > always {
> > > > > > // Fallback option: If standard cleanWs() stalls out on
> > > > > permissions
> > > > > > script {
> > > > > > try {
> > > > > > cleanWs deleteDirs: true, disableDeferredWipeout:
> > > true
> > > > > > } catch (Exception e) {
> > > > > > echo “Standard cleanup failed, launching Docker Root
> > > > > > Purge force-override...”
> > > > > > sh “”"
> > > > > > docker run --rm \
> > > > > > -v ${WORKSPACE}/..:/zap \
> > > > > > alpine:latest \
> > > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > > > > > “”"
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > > }
> > > > > >
> > > > > > ---------------------------------------------------------------------
> > > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > > >
> > > > > >
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by Bob Thomson <bt...@apache.org>.
Like I say, not my code as such, a Gen AI suggestion, but I would say it's an easy way for the process running as the jenkins user, to remove files which are not owned by the jenkins user, which seems to often be the reason for these ws-cleanup directories not being removed (Often in the Camel project case anyway). The other reason might be a restart of the server for example, which doesn't happen on lucene but sometimes does happen on some other CI servers we have due to Jenkins locking up (GC/RAM usage often).
On 2026/07/14 12:49:48 David Smiley wrote:
> I modified one build,
> https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ to
> have an additional shell execution step with that snippet, which performs
> the cleanup via Docker as you suggested, albeit not with the Jenkins/groovy
> wrapper all around it. It ran successfully this morning. I'm not sure why
> Docker is involved for what looks like a simple file removal; might you
> explain?
>
> On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <bt...@apache.org> wrote:
>
> > Hi,
> >
> > In Infra, we don't work on job definitions as that is a project's domain.
> > The above is only a suggestion which may or may not help.
> > If it is easier, you could look at adding similar code to the jobs as a
> > shell script post build step in order to remove the files. Aurélien Pupier
> > of the Camel project has been looking at the same issue there, so worth
> > having a chat with them about potential solutions.
> >
> > In terms of the jobs which are leaving ws-cleanup directories behind, it
> > seems to be many of them. On lucene 3 today, there are already 113 such
> > directories, 213GB so far.
> >
> > the jobs that they are associated with, from per their directory names are:
> >
> > OWASP-9.x
> > OWASP-main
> > Solr-Artifacts-10.x
> > Solr-Artifacts-9.11
> > Solr-Artifacts-9.x
> > Solr-Artifacts-main
> > Solr-BadApples-Tests-main
> > Solr-Docker-Nightly-9.11
> > Solr-Lint-10.x
> > Solr-Lint-9.11
> > Solr-Lint-9.x
> > Solr-Lint-main
> > Solr-Maven-Snapshots-main
> > Solr-Test-10.0
> > Solr-Test-10.x
> > Solr-Test-9.11
> > Solr-Test-9.x
> > Solr-Test-main
> > Solr-TestIntegration-10.x
> > Solr-TestIntegration-9.11
> > Solr-TestIntegration-9.x
> > Solr-TestIntegration-main
> > solr-desktop-ui-nightly
> >
> > On 4, there are 80 from these builds:
> >
> > OWASP-9.x
> > OWASP-main
> > Solr-Artifacts-10.x
> > Solr-Artifacts-9.11
> > Solr-Artifacts-9.x
> > Solr-Docker-Nightly-9.11
> > Solr-Lint-10.x
> > Solr-Lint-9.11
> > Solr-Lint-9.x
> > Solr-Lint-main
> > Solr-Maven-Snapshots-main
> > Solr-Test-10.0
> > Solr-Test-10.x
> > Solr-Test-9.11
> > Solr-Test-9.x
> > Solr-Test-main
> > Solr-TestIntegration-9.11
> > Solr-TestIntegration-9.x
> >
> > On 2026/07/03 20:32:09 David Smiley wrote:
> > > I know too little about Jenkins to know how to apply this solution, but I
> > > can try to help. The Solr jobs are defined in Jenkins, not in our repo
> > in
> > > Jenkinsfile. Maybe you could apply this to one of the jobs yourself?
> > Does
> > > this need to be done on all jobs or just some or will just one do?
> > >
> > > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org> wrote:
> > >
> > > > This has come up as an issue again today, necessitating intervention to
> > > > keep the service available.
> > > >
> > > > On 2026/06/29 07:04:23 Robert Thomson wrote:
> > > > > Hi,
> > > > >
> > > > > We are regularly noticing a build-up of large ws-cleanup directories
> > > > > on lucene3 and 4, necessitating deletion. Camel also experiences this
> > > > > issue with builds and the root cause is that there are files created
> > > > > as part of the docker build that are not owned by the user used for
> > > > > the cleanup, so fail to be deleted by the workspace clean up plugin.
> > > > >
> > > > > Searching with GenAI came up with the following suggestion when
> > > > > discussing with them that you may like to try out in your pipelines
> > > > > that create docker images.
> > > > >
> > > > > Kind regards,
> > > > > -Bob Thomson,
> > > > > ASF Infrastructure
> > > > >
> > > > > post {
> > > > > always {
> > > > > // Fallback option: If standard cleanWs() stalls out on
> > > > permissions
> > > > > script {
> > > > > try {
> > > > > cleanWs deleteDirs: true, disableDeferredWipeout:
> > true
> > > > > } catch (Exception e) {
> > > > > echo “Standard cleanup failed, launching Docker Root
> > > > > Purge force-override...”
> > > > > sh “”"
> > > > > docker run --rm \
> > > > > -v ${WORKSPACE}/..:/zap \
> > > > > alpine:latest \
> > > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > > > > “”"
> > > > > }
> > > > > }
> > > > > }
> > > > > }
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by David Smiley <ds...@apache.org>.
I modified one build,
https://ci-builds.apache.org/job/Solr/job/Solr-Maven-Snapshots-main/ to
have an additional shell execution step with that snippet, which performs
the cleanup via Docker as you suggested, albeit not with the Jenkins/groovy
wrapper all around it. It ran successfully this morning. I'm not sure why
Docker is involved for what looks like a simple file removal; might you
explain?
On Mon, Jul 6, 2026 at 8:41 AM Bob Thomson <bt...@apache.org> wrote:
> Hi,
>
> In Infra, we don't work on job definitions as that is a project's domain.
> The above is only a suggestion which may or may not help.
> If it is easier, you could look at adding similar code to the jobs as a
> shell script post build step in order to remove the files. Aurélien Pupier
> of the Camel project has been looking at the same issue there, so worth
> having a chat with them about potential solutions.
>
> In terms of the jobs which are leaving ws-cleanup directories behind, it
> seems to be many of them. On lucene 3 today, there are already 113 such
> directories, 213GB so far.
>
> the jobs that they are associated with, from per their directory names are:
>
> OWASP-9.x
> OWASP-main
> Solr-Artifacts-10.x
> Solr-Artifacts-9.11
> Solr-Artifacts-9.x
> Solr-Artifacts-main
> Solr-BadApples-Tests-main
> Solr-Docker-Nightly-9.11
> Solr-Lint-10.x
> Solr-Lint-9.11
> Solr-Lint-9.x
> Solr-Lint-main
> Solr-Maven-Snapshots-main
> Solr-Test-10.0
> Solr-Test-10.x
> Solr-Test-9.11
> Solr-Test-9.x
> Solr-Test-main
> Solr-TestIntegration-10.x
> Solr-TestIntegration-9.11
> Solr-TestIntegration-9.x
> Solr-TestIntegration-main
> solr-desktop-ui-nightly
>
> On 4, there are 80 from these builds:
>
> OWASP-9.x
> OWASP-main
> Solr-Artifacts-10.x
> Solr-Artifacts-9.11
> Solr-Artifacts-9.x
> Solr-Docker-Nightly-9.11
> Solr-Lint-10.x
> Solr-Lint-9.11
> Solr-Lint-9.x
> Solr-Lint-main
> Solr-Maven-Snapshots-main
> Solr-Test-10.0
> Solr-Test-10.x
> Solr-Test-9.11
> Solr-Test-9.x
> Solr-Test-main
> Solr-TestIntegration-9.11
> Solr-TestIntegration-9.x
>
> On 2026/07/03 20:32:09 David Smiley wrote:
> > I know too little about Jenkins to know how to apply this solution, but I
> > can try to help. The Solr jobs are defined in Jenkins, not in our repo
> in
> > Jenkinsfile. Maybe you could apply this to one of the jobs yourself?
> Does
> > this need to be done on all jobs or just some or will just one do?
> >
> > On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org> wrote:
> >
> > > This has come up as an issue again today, necessitating intervention to
> > > keep the service available.
> > >
> > > On 2026/06/29 07:04:23 Robert Thomson wrote:
> > > > Hi,
> > > >
> > > > We are regularly noticing a build-up of large ws-cleanup directories
> > > > on lucene3 and 4, necessitating deletion. Camel also experiences this
> > > > issue with builds and the root cause is that there are files created
> > > > as part of the docker build that are not owned by the user used for
> > > > the cleanup, so fail to be deleted by the workspace clean up plugin.
> > > >
> > > > Searching with GenAI came up with the following suggestion when
> > > > discussing with them that you may like to try out in your pipelines
> > > > that create docker images.
> > > >
> > > > Kind regards,
> > > > -Bob Thomson,
> > > > ASF Infrastructure
> > > >
> > > > post {
> > > > always {
> > > > // Fallback option: If standard cleanWs() stalls out on
> > > permissions
> > > > script {
> > > > try {
> > > > cleanWs deleteDirs: true, disableDeferredWipeout:
> true
> > > > } catch (Exception e) {
> > > > echo “Standard cleanup failed, launching Docker Root
> > > > Purge force-override...”
> > > > sh “”"
> > > > docker run --rm \
> > > > -v ${WORKSPACE}/..:/zap \
> > > > alpine:latest \
> > > > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > > > “”"
> > > > }
> > > > }
> > > > }
> > > > }
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > > For additional commands, e-mail: dev-help@solr.apache.org
> > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by Bob Thomson <bt...@apache.org>.
Hi,
In Infra, we don't work on job definitions as that is a project's domain. The above is only a suggestion which may or may not help.
If it is easier, you could look at adding similar code to the jobs as a shell script post build step in order to remove the files. Aurélien Pupier of the Camel project has been looking at the same issue there, so worth having a chat with them about potential solutions.
In terms of the jobs which are leaving ws-cleanup directories behind, it seems to be many of them. On lucene 3 today, there are already 113 such directories, 213GB so far.
the jobs that they are associated with, from per their directory names are:
OWASP-9.x
OWASP-main
Solr-Artifacts-10.x
Solr-Artifacts-9.11
Solr-Artifacts-9.x
Solr-Artifacts-main
Solr-BadApples-Tests-main
Solr-Docker-Nightly-9.11
Solr-Lint-10.x
Solr-Lint-9.11
Solr-Lint-9.x
Solr-Lint-main
Solr-Maven-Snapshots-main
Solr-Test-10.0
Solr-Test-10.x
Solr-Test-9.11
Solr-Test-9.x
Solr-Test-main
Solr-TestIntegration-10.x
Solr-TestIntegration-9.11
Solr-TestIntegration-9.x
Solr-TestIntegration-main
solr-desktop-ui-nightly
On 4, there are 80 from these builds:
OWASP-9.x
OWASP-main
Solr-Artifacts-10.x
Solr-Artifacts-9.11
Solr-Artifacts-9.x
Solr-Docker-Nightly-9.11
Solr-Lint-10.x
Solr-Lint-9.11
Solr-Lint-9.x
Solr-Lint-main
Solr-Maven-Snapshots-main
Solr-Test-10.0
Solr-Test-10.x
Solr-Test-9.11
Solr-Test-9.x
Solr-Test-main
Solr-TestIntegration-9.11
Solr-TestIntegration-9.x
On 2026/07/03 20:32:09 David Smiley wrote:
> I know too little about Jenkins to know how to apply this solution, but I
> can try to help. The Solr jobs are defined in Jenkins, not in our repo in
> Jenkinsfile. Maybe you could apply this to one of the jobs yourself? Does
> this need to be done on all jobs or just some or will just one do?
>
> On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org> wrote:
>
> > This has come up as an issue again today, necessitating intervention to
> > keep the service available.
> >
> > On 2026/06/29 07:04:23 Robert Thomson wrote:
> > > Hi,
> > >
> > > We are regularly noticing a build-up of large ws-cleanup directories
> > > on lucene3 and 4, necessitating deletion. Camel also experiences this
> > > issue with builds and the root cause is that there are files created
> > > as part of the docker build that are not owned by the user used for
> > > the cleanup, so fail to be deleted by the workspace clean up plugin.
> > >
> > > Searching with GenAI came up with the following suggestion when
> > > discussing with them that you may like to try out in your pipelines
> > > that create docker images.
> > >
> > > Kind regards,
> > > -Bob Thomson,
> > > ASF Infrastructure
> > >
> > > post {
> > > always {
> > > // Fallback option: If standard cleanWs() stalls out on
> > permissions
> > > script {
> > > try {
> > > cleanWs deleteDirs: true, disableDeferredWipeout: true
> > > } catch (Exception e) {
> > > echo “Standard cleanup failed, launching Docker Root
> > > Purge force-override...”
> > > sh “”"
> > > docker run --rm \
> > > -v ${WORKSPACE}/..:/zap \
> > > alpine:latest \
> > > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > > “”"
> > > }
> > > }
> > > }
> > > }
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > > For additional commands, e-mail: dev-help@solr.apache.org
> > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by David Smiley <ds...@apache.org>.
I know too little about Jenkins to know how to apply this solution, but I
can try to help. The Solr jobs are defined in Jenkins, not in our repo in
Jenkinsfile. Maybe you could apply this to one of the jobs yourself? Does
this need to be done on all jobs or just some or will just one do?
On Fri, Jul 3, 2026 at 8:20 AM Bob Thomson <bt...@apache.org> wrote:
> This has come up as an issue again today, necessitating intervention to
> keep the service available.
>
> On 2026/06/29 07:04:23 Robert Thomson wrote:
> > Hi,
> >
> > We are regularly noticing a build-up of large ws-cleanup directories
> > on lucene3 and 4, necessitating deletion. Camel also experiences this
> > issue with builds and the root cause is that there are files created
> > as part of the docker build that are not owned by the user used for
> > the cleanup, so fail to be deleted by the workspace clean up plugin.
> >
> > Searching with GenAI came up with the following suggestion when
> > discussing with them that you may like to try out in your pipelines
> > that create docker images.
> >
> > Kind regards,
> > -Bob Thomson,
> > ASF Infrastructure
> >
> > post {
> > always {
> > // Fallback option: If standard cleanWs() stalls out on
> permissions
> > script {
> > try {
> > cleanWs deleteDirs: true, disableDeferredWipeout: true
> > } catch (Exception e) {
> > echo “Standard cleanup failed, launching Docker Root
> > Purge force-override...”
> > sh “”"
> > docker run --rm \
> > -v ${WORKSPACE}/..:/zap \
> > alpine:latest \
> > sh -c ‘rm -rf /zap/*-ws-cleanup*’
> > “”"
> > }
> > }
> > }
> > }
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> > For additional commands, e-mail: dev-help@solr.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
Re: ACTION NEEDED - lucene3/4 disk space and workspace cleanup files
Posted by Bob Thomson <bt...@apache.org>.
This has come up as an issue again today, necessitating intervention to keep the service available.
On 2026/06/29 07:04:23 Robert Thomson wrote:
> Hi,
>
> We are regularly noticing a build-up of large ws-cleanup directories
> on lucene3 and 4, necessitating deletion. Camel also experiences this
> issue with builds and the root cause is that there are files created
> as part of the docker build that are not owned by the user used for
> the cleanup, so fail to be deleted by the workspace clean up plugin.
>
> Searching with GenAI came up with the following suggestion when
> discussing with them that you may like to try out in your pipelines
> that create docker images.
>
> Kind regards,
> -Bob Thomson,
> ASF Infrastructure
>
> post {
> always {
> // Fallback option: If standard cleanWs() stalls out on permissions
> script {
> try {
> cleanWs deleteDirs: true, disableDeferredWipeout: true
> } catch (Exception e) {
> echo “Standard cleanup failed, launching Docker Root
> Purge force-override...”
> sh “”"
> docker run --rm \
> -v ${WORKSPACE}/..:/zap \
> alpine:latest \
> sh -c ‘rm -rf /zap/*-ws-cleanup*’
> “”"
> }
> }
> }
> }
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
> For additional commands, e-mail: dev-help@solr.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@solr.apache.org
For additional commands, e-mail: dev-help@solr.apache.org