You are viewing a plain text version of this content. The canonical link for it is here.
Posted to builds@apache.org by Uwe Schindler <us...@apache.org> on 2011/03/19 13:10:20 UTC
No diskspace on Lucene Jail
Hi,
the Lucene Jail has no disk space anymore. As the filesystem is shared with
other jails, there seems to be something on another jail using all free disk
space.
[uschindler@lucene ~]$ df -h
Filesystem Size Used Avail Capacity
Mounted on
zroot/jails/lucene.zones.apache.org 66G 66G 0B 100% /
Jenkins of cource fails to build.
Uwe
-----
Uwe Schindler
uschindler@apache.org
Apache Lucene PMC Member / Committer
Bremen, Germany
http://lucene.apache.org/
Re: No diskspace on Lucene Jail
Posted by Mads Toftum <ma...@toftum.dk>.
On Sat, Mar 19, 2011 at 01:59:28PM +0100, Uwe Schindler wrote:
> I am unable to delete any files on this machine:
>
> [root@lucene /home/hudson/hudson-slave/workspace]# rm -rf *
> rm: Lucene-3.x/artifacts/changes/ChangesFixedWidthStyle.css: No space left
> on device
> rm: Lucene-3.x/artifacts/changes/ChangesSimpleStyle.css: No space left on
> device
> rm: Lucene-3.x/artifacts/changes/ChangesFancyStyle.css: No space left on
> device
> rm: Lucene-3.x/artifacts/changes/Changes.html: No space left on device
> rm: Lucene-3.x/artifacts/changes/Contrib-Changes.html: No space left on
> device
>
When that happens, the easy fix is to truncate large files first
(cat /dev/null > file).
vh
Mads Toftum
--
http://soulfood.dk
RE: No diskspace on Lucene Jail
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Gav,
I am unable to delete any files on this machine:
[root@lucene /home/hudson/hudson-slave/workspace]# rm -rf *
rm: Lucene-3.x/artifacts/changes/ChangesFixedWidthStyle.css: No space left
on device
rm: Lucene-3.x/artifacts/changes/ChangesSimpleStyle.css: No space left on
device
rm: Lucene-3.x/artifacts/changes/ChangesFancyStyle.css: No space left on
device
rm: Lucene-3.x/artifacts/changes/Changes.html: No space left on device
rm: Lucene-3.x/artifacts/changes/Contrib-Changes.html: No space left on
device
It deletes some files (its verified that it does, they disappear!) but one
minute later again its 100% full.
We had this issue before, too. At this time somebody told me that the free
space is shared between multiple JAILs. So for me it looks like I delete the
files and some other process in some other jail fills up the freed space as
soon as it gets free.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Gav... [mailto:gavin@16degrees.com.au]
> Sent: Saturday, March 19, 2011 1:31 PM
> To: builds@apache.org
> Subject: RE: No diskspace on Lucene Jail
>
>
>
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uschindler@apache.org]
> > Sent: Saturday, 19 March 2011 10:10 PM
> > To: builds@apache.org
> > Subject: No diskspace on Lucene Jail
> >
> > Hi,
> >
> > the Lucene Jail has no disk space anymore. As the filesystem is shared
> with
> > other jails, there seems to be something on another jail using all
> > free
> disk
> > space.
>
> erm no, as shown by your df- h output below Lucene is using 66GB, already.
>
> The next biggest jail is Gump with 36GB, nearly 1/2 of what Lucene is
using.
> The rest all hover between mostly 1 or 2GB up to 19GB.
>
> The point is until our array arrives, space is short. You need to trim
your logs,
> do something to reduce your current usage until then - the other are doing
it
> and and revisit those jails again tomorrow as well as yours. Best thing to
do is
> clean up what you can, cron what you can to do this until our array turns
up
> which could be a month or more.
>
> >
> > [uschindler@lucene ~]$ df -h
> > Filesystem Size Used Avail Capacity
> > Mounted on
> > zroot/jails/lucene.zones.apache.org 66G 66G 0B 100% /
> >
> > Jenkins of cource fails to build.
>
> Of course.
>
> Gav...
>
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > uschindler@apache.org
> > Apache Lucene PMC Member / Committer
> > Bremen, Germany
> > http://lucene.apache.org/
> >
>
>
RE: No diskspace on Lucene Jail
Posted by "Gav..." <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Sunday, 20 March 2011 7:34 AM
> To: builds@apache.org
> Subject: RE: No diskspace on Lucene Jail
>
> It's like Truman Show :-)
Explain?
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Gav... [mailto:gavin@16degrees.com.au]
> > Sent: Saturday, March 19, 2011 10:02 PM
> > To: builds@apache.org
> > Subject: RE: No diskspace on Lucene Jail
> >
> >
> >
> > > -----Original Message-----
> > > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > > Sent: Saturday, 19 March 2011 11:09 PM
> > > To: builds@apache.org
> > > Subject: RE: No diskspace on Lucene Jail
> > >
> > > > 52GB is in /home/hudson/hudson-slave/workspace
> > > >
> > > > so would help to clean up in there and tighten up on how many
> > > > builds to keep etc...
> > > >
> > > > thanks
> > >
> > > No builds are kept there, they are copied to hudson master after
> finished.
> > > Workspace only contains the current build's workspace and is cleaned
> > > up by most jobs before build.
> >
> > I'm not sure this is happening, you say the workspace is cleaned up
> > yet I
> just
> > removed 52GB from there. Is there something we need to check? Should
> > source checkouts be clobbered each time, for example.
> >
> > Thanks
> >
> > Gav...
> >
> > >
> > > Uwe
> >
> >
>
RE: No diskspace on Lucene Jail
Posted by Uwe Schindler <uw...@thetaphi.de>.
It's like Truman Show :-)
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Gav... [mailto:gavin@16degrees.com.au]
> Sent: Saturday, March 19, 2011 10:02 PM
> To: builds@apache.org
> Subject: RE: No diskspace on Lucene Jail
>
>
>
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uwe@thetaphi.de]
> > Sent: Saturday, 19 March 2011 11:09 PM
> > To: builds@apache.org
> > Subject: RE: No diskspace on Lucene Jail
> >
> > > 52GB is in /home/hudson/hudson-slave/workspace
> > >
> > > so would help to clean up in there and tighten up on how many builds
> > > to keep etc...
> > >
> > > thanks
> >
> > No builds are kept there, they are copied to hudson master after
finished.
> > Workspace only contains the current build's workspace and is cleaned
> > up by most jobs before build.
>
> I'm not sure this is happening, you say the workspace is cleaned up yet I
just
> removed 52GB from there. Is there something we need to check? Should
> source checkouts be clobbered each time, for example.
>
> Thanks
>
> Gav...
>
> >
> > Uwe
>
>
RE: No diskspace on Lucene Jail
Posted by "Gav..." <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Saturday, 19 March 2011 11:09 PM
> To: builds@apache.org
> Subject: RE: No diskspace on Lucene Jail
>
> > 52GB is in /home/hudson/hudson-slave/workspace
> >
> > so would help to clean up in there and tighten up on how many builds
> > to keep etc...
> >
> > thanks
>
> No builds are kept there, they are copied to hudson master after finished.
> Workspace only contains the current build's workspace and is cleaned up by
> most jobs before build.
I'm not sure this is happening, you say the workspace is cleaned up yet I
just
removed 52GB from there. Is there something we need to check? Should
source checkouts be clobbered each time, for example.
Thanks
Gav...
>
> Uwe
RE: No diskspace on Lucene Jail
Posted by Uwe Schindler <uw...@thetaphi.de>.
> 52GB is in /home/hudson/hudson-slave/workspace
>
> so would help to clean up in there and tighten up on how many builds to
> keep etc...
>
> thanks
No builds are kept there, they are copied to hudson master after finished.
Workspace only contains the current build's workspace and is cleaned up by
most jobs before build.
Uwe
RE: No diskspace on Lucene Jail
Posted by "Gav..." <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Gav... [mailto:gavin@16degrees.com.au]
> Sent: Saturday, 19 March 2011 10:31 PM
> To: builds@apache.org
> Subject: RE: No diskspace on Lucene Jail
>
>
>
> > -----Original Message-----
> > From: Uwe Schindler [mailto:uschindler@apache.org]
> > Sent: Saturday, 19 March 2011 10:10 PM
> > To: builds@apache.org
> > Subject: No diskspace on Lucene Jail
> >
> > Hi,
> >
> > the Lucene Jail has no disk space anymore. As the filesystem is shared
> with
> > other jails, there seems to be something on another jail using all
> > free
> disk
> > space.
>
> erm no, as shown by your df- h output below Lucene is using 66GB, already.
52GB is in /home/hudson/hudson-slave/workspace
so would help to clean up in there and tighten up on how many builds
to keep etc...
thanks
Gav...
>
> The next biggest jail is Gump with 36GB, nearly 1/2 of what Lucene is
using.
> The rest all hover between mostly 1 or 2GB up to 19GB.
>
> The point is until our array arrives, space is short. You need to trim
your logs,
> do something to reduce your current usage until then - the other are doing
it
> and and revisit those jails again tomorrow as well as yours. Best thing to
do is
> clean up what you can, cron what you can to do this until our array turns
up
> which could be a month or more.
>
> >
> > [uschindler@lucene ~]$ df -h
> > Filesystem Size Used Avail Capacity
> > Mounted on
> > zroot/jails/lucene.zones.apache.org 66G 66G 0B 100% /
> >
> > Jenkins of cource fails to build.
>
> Of course.
>
> Gav...
>
> >
> > Uwe
> >
> > -----
> > Uwe Schindler
> > uschindler@apache.org
> > Apache Lucene PMC Member / Committer
> > Bremen, Germany
> > http://lucene.apache.org/
> >
>
>
RE: No diskspace on Lucene Jail
Posted by "Gav..." <ga...@16degrees.com.au>.
> -----Original Message-----
> From: Uwe Schindler [mailto:uschindler@apache.org]
> Sent: Saturday, 19 March 2011 10:10 PM
> To: builds@apache.org
> Subject: No diskspace on Lucene Jail
>
> Hi,
>
> the Lucene Jail has no disk space anymore. As the filesystem is shared
with
> other jails, there seems to be something on another jail using all free
disk
> space.
erm no, as shown by your df- h output below Lucene is using 66GB, already.
The next biggest jail is Gump with 36GB, nearly 1/2 of what Lucene is using.
The rest all hover between mostly 1 or 2GB up to 19GB.
The point is until our array arrives, space is short. You need to trim your
logs,
do something to reduce your current usage until then - the other are doing
it and and revisit those jails again tomorrow as well as yours. Best thing
to
do is clean up what you can, cron what you can to do this until our array
turns
up which could be a month or more.
>
> [uschindler@lucene ~]$ df -h
> Filesystem Size Used Avail Capacity
> Mounted on
> zroot/jails/lucene.zones.apache.org 66G 66G 0B 100% /
>
> Jenkins of cource fails to build.
Of course.
Gav...
>
> Uwe
>
> -----
> Uwe Schindler
> uschindler@apache.org
> Apache Lucene PMC Member / Committer
> Bremen, Germany
> http://lucene.apache.org/
>