You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@jackrabbit.apache.org by Bill Oconnor <bo...@plos.org> on 2010/03/22 19:11:12 UTC
Disk Full
Hello,
I have a standalone version of JR 2.0 on an EC2 instance with CENTOS 5.
I have a 750 GB volume that I launch JR on. I created approximately 17K
directories which I plan to unzip approx 100K different files. The total
space consumed should not be more that 500 GB.
I mount the webdav volume using fuse-davfs2. The directories are all
made but my script stops unzipping and copying files at around 13 GB.
The volume itself only reports to be 26 GB.
Anybody know what is going on here? Is this a JR or fuse-davfs
configuration? Is there some magic to mounting it? Searched the list but
haven't identified anything that answers my questions.
Thanks,
Bill OConnor
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This email is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your
system. Any unauthorized use, disclosure or copying is not permitted. The views or opinions presented are solely those of the sender and do
not necessarily represent those of Public Library of Science unless otherwise specifically stated. Please note that neither Public Library
of Science nor any of its agents accept any responsibility for any viruses that may be contained in this e-mail or its attachments and it
is your responsibility to scan the e-mail and attachments (if any).
Re: Disk Full
Posted by Bill Oconnor <bo...@plos.org>.
I think it is a temp dir thing. My root partition is at a %100 and JR
hung. What files are being opened by jackrabbit?
Will -Djava.io.tmpdir=/repo/temp in the JR startup command move them
there?
On Mon, 2010-03-22 at 15:36 -0700, Dragisa Krsmanovic wrote:
> What does df -i tell you ?
>
> Other possibility is that, for some reason, you exceede
http://localhost:8080/repository/default
1666666 -1 1666667 - /plos_repo
> d max open files.
> What does ulimit -a tell you ?
ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 61504
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 61504
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
>
> lsof | wc -l will give you how many open files you have.
>
>
> On Mon, 2010-03-22 at 15:08 -0700, Bill Oconnor wrote:
> > Hi, Thanks for getting back to me.
> >
> > I don't think it is a problem with EC2. S3 is more like a backup service
> > I'm working with network attached disk volumes.
> >
> > I created a 750 gig volume which I mount onto the instance. The original
> > data that I am using is on a similar volume (500 gig). I start the
> > standalone JR from the directory on the 750 gig volume and the
> > repository gets created there so there is plenty of space.
> >
> > I installed fuse-davfs2 so that I can mount webdav volume locally:
> >
> > mount -t davfs localhost:8080/repository/default /repo/plos_repo
> >
> > I use the vanilla configuration of JR.
> >
> > At this point everything is fine. I create a directory
> > called /repo/plos_repo/10.1371 and then make around 17000
> > subdirectories. My script then proceeds to unzip into the appropriate
> > directories.
> >
> > The script starts failing after around 300 zip files are processed. If I
> > do a df -h there seems to be plenty of room on the volumes.
> >
> > If I try and do the process manually at this point by unzipping the file
> > directly to the mounted repository the directory it fails.
> >
> > I'm feeling like there is a configuration issue but I haven't seen any
> > docs that specifically address this.
> >
> > ~bill
> >
> > On Mon, 2010-03-22 at 20:12 +0100, Paul PODGORSEK wrote:
> > > Hello Bill,
> > >
> > > Are you sure that the problem isn't coming from the EC2 instance?
> > > I have read some information about it, and it seems the filesystem that is
> > > used (S3) isn't a standard Posix one.
> > > For example, S3 stores objects up to 5 gigabytes in size.
> > >
> > > Two links that may help:
> > > http://en.wikipedia.org/wiki/Amazon_EC2
> > > http://en.wikipedia.org/wiki/Amazon_S3
> > >
> > > Best regards,
> > > Paul.
> > >
> > > --------------------------------------------------
> > > From: "Bill Oconnor" <bo...@plos.org>
> > > Sent: Monday, March 22, 2010 7:11 PM
> > > To: <us...@jackrabbit.apache.org>
> > > Subject: Disk Full
> > >
> > > > Hello,
> > > >
> > > > I have a standalone version of JR 2.0 on an EC2 instance with CENTOS 5.
> > > > I have a 750 GB volume that I launch JR on. I created approximately 17K
> > > > directories which I plan to unzip approx 100K different files. The total
> > > > space consumed should not be more that 500 GB.
> > > >
> > > > I mount the webdav volume using fuse-davfs2. The directories are all
> > > > made but my script stops unzipping and copying files at around 13 GB.
> > > > The volume itself only reports to be 26 GB.
> > > >
> > > > Anybody know what is going on here? Is this a JR or fuse-davfs
> > > > configuration? Is there some magic to mounting it? Searched the list but
> > > > haven't identified anything that answers my questions.
> > > >
> > > > Thanks,
> > > > Bill OConnor
> > > >
> > > >
> > > >
> >
> >
>
>
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This email is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your
system. Any unauthorized use, disclosure or copying is not permitted. The views or opinions presented are solely those of the sender and do
not necessarily represent those of Public Library of Science unless otherwise specifically stated. Please note that neither Public Library
of Science nor any of its agents accept any responsibility for any viruses that may be contained in this e-mail or its attachments and it
is your responsibility to scan the e-mail and attachments (if any).
Re: Disk Full
Posted by Dragisa Krsmanovic <dk...@plos.org>.
What does df -i tell you ?
Other possibility is that, for some reason, you exceeded max open files.
What does ulimit -a tell you ?
lsof | wc -l will give you how many open files you have.
On Mon, 2010-03-22 at 15:08 -0700, Bill Oconnor wrote:
> Hi, Thanks for getting back to me.
>
> I don't think it is a problem with EC2. S3 is more like a backup service
> I'm working with network attached disk volumes.
>
> I created a 750 gig volume which I mount onto the instance. The original
> data that I am using is on a similar volume (500 gig). I start the
> standalone JR from the directory on the 750 gig volume and the
> repository gets created there so there is plenty of space.
>
> I installed fuse-davfs2 so that I can mount webdav volume locally:
>
> mount -t davfs localhost:8080/repository/default /repo/plos_repo
>
> I use the vanilla configuration of JR.
>
> At this point everything is fine. I create a directory
> called /repo/plos_repo/10.1371 and then make around 17000
> subdirectories. My script then proceeds to unzip into the appropriate
> directories.
>
> The script starts failing after around 300 zip files are processed. If I
> do a df -h there seems to be plenty of room on the volumes.
>
> If I try and do the process manually at this point by unzipping the file
> directly to the mounted repository the directory it fails.
>
> I'm feeling like there is a configuration issue but I haven't seen any
> docs that specifically address this.
>
> ~bill
>
> On Mon, 2010-03-22 at 20:12 +0100, Paul PODGORSEK wrote:
> > Hello Bill,
> >
> > Are you sure that the problem isn't coming from the EC2 instance?
> > I have read some information about it, and it seems the filesystem that is
> > used (S3) isn't a standard Posix one.
> > For example, S3 stores objects up to 5 gigabytes in size.
> >
> > Two links that may help:
> > http://en.wikipedia.org/wiki/Amazon_EC2
> > http://en.wikipedia.org/wiki/Amazon_S3
> >
> > Best regards,
> > Paul.
> >
> > --------------------------------------------------
> > From: "Bill Oconnor" <bo...@plos.org>
> > Sent: Monday, March 22, 2010 7:11 PM
> > To: <us...@jackrabbit.apache.org>
> > Subject: Disk Full
> >
> > > Hello,
> > >
> > > I have a standalone version of JR 2.0 on an EC2 instance with CENTOS 5.
> > > I have a 750 GB volume that I launch JR on. I created approximately 17K
> > > directories which I plan to unzip approx 100K different files. The total
> > > space consumed should not be more that 500 GB.
> > >
> > > I mount the webdav volume using fuse-davfs2. The directories are all
> > > made but my script stops unzipping and copying files at around 13 GB.
> > > The volume itself only reports to be 26 GB.
> > >
> > > Anybody know what is going on here? Is this a JR or fuse-davfs
> > > configuration? Is there some magic to mounting it? Searched the list but
> > > haven't identified anything that answers my questions.
> > >
> > > Thanks,
> > > Bill OConnor
> > >
> > >
> > >
>
>
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This email is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your
system. Any unauthorized use, disclosure or copying is not permitted. The views or opinions presented are solely those of the sender and do
not necessarily represent those of Public Library of Science unless otherwise specifically stated. Please note that neither Public Library
of Science nor any of its agents accept any responsibility for any viruses that may be contained in this e-mail or its attachments and it
is your responsibility to scan the e-mail and attachments (if any).
Re: Disk Full
Posted by Bill Oconnor <bo...@plos.org>.
Hi, Thanks for getting back to me.
I don't think it is a problem with EC2. S3 is more like a backup service
I'm working with network attached disk volumes.
I created a 750 gig volume which I mount onto the instance. The original
data that I am using is on a similar volume (500 gig). I start the
standalone JR from the directory on the 750 gig volume and the
repository gets created there so there is plenty of space.
I installed fuse-davfs2 so that I can mount webdav volume locally:
mount -t davfs localhost:8080/repository/default /repo/plos_repo
I use the vanilla configuration of JR.
At this point everything is fine. I create a directory
called /repo/plos_repo/10.1371 and then make around 17000
subdirectories. My script then proceeds to unzip into the appropriate
directories.
The script starts failing after around 300 zip files are processed. If I
do a df -h there seems to be plenty of room on the volumes.
If I try and do the process manually at this point by unzipping the file
directly to the mounted repository the directory it fails.
I'm feeling like there is a configuration issue but I haven't seen any
docs that specifically address this.
~bill
On Mon, 2010-03-22 at 20:12 +0100, Paul PODGORSEK wrote:
> Hello Bill,
>
> Are you sure that the problem isn't coming from the EC2 instance?
> I have read some information about it, and it seems the filesystem that is
> used (S3) isn't a standard Posix one.
> For example, S3 stores objects up to 5 gigabytes in size.
>
> Two links that may help:
> http://en.wikipedia.org/wiki/Amazon_EC2
> http://en.wikipedia.org/wiki/Amazon_S3
>
> Best regards,
> Paul.
>
> --------------------------------------------------
> From: "Bill Oconnor" <bo...@plos.org>
> Sent: Monday, March 22, 2010 7:11 PM
> To: <us...@jackrabbit.apache.org>
> Subject: Disk Full
>
> > Hello,
> >
> > I have a standalone version of JR 2.0 on an EC2 instance with CENTOS 5.
> > I have a 750 GB volume that I launch JR on. I created approximately 17K
> > directories which I plan to unzip approx 100K different files. The total
> > space consumed should not be more that 500 GB.
> >
> > I mount the webdav volume using fuse-davfs2. The directories are all
> > made but my script stops unzipping and copying files at around 13 GB.
> > The volume itself only reports to be 26 GB.
> >
> > Anybody know what is going on here? Is this a JR or fuse-davfs
> > configuration? Is there some magic to mounting it? Searched the list but
> > haven't identified anything that answers my questions.
> >
> > Thanks,
> > Bill OConnor
> >
> >
> >
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This email is confidential to the intended recipient. If you have received it in error, please notify the sender and delete it from your
system. Any unauthorized use, disclosure or copying is not permitted. The views or opinions presented are solely those of the sender and do
not necessarily represent those of Public Library of Science unless otherwise specifically stated. Please note that neither Public Library
of Science nor any of its agents accept any responsibility for any viruses that may be contained in this e-mail or its attachments and it
is your responsibility to scan the e-mail and attachments (if any).
Re: Disk Full
Posted by Paul PODGORSEK <Th...@hotmail.com>.
Hello Bill,
Are you sure that the problem isn't coming from the EC2 instance?
I have read some information about it, and it seems the filesystem that is
used (S3) isn't a standard Posix one.
For example, S3 stores objects up to 5 gigabytes in size.
Two links that may help:
http://en.wikipedia.org/wiki/Amazon_EC2
http://en.wikipedia.org/wiki/Amazon_S3
Best regards,
Paul.
--------------------------------------------------
From: "Bill Oconnor" <bo...@plos.org>
Sent: Monday, March 22, 2010 7:11 PM
To: <us...@jackrabbit.apache.org>
Subject: Disk Full
> Hello,
>
> I have a standalone version of JR 2.0 on an EC2 instance with CENTOS 5.
> I have a 750 GB volume that I launch JR on. I created approximately 17K
> directories which I plan to unzip approx 100K different files. The total
> space consumed should not be more that 500 GB.
>
> I mount the webdav volume using fuse-davfs2. The directories are all
> made but my script stops unzipping and copying files at around 13 GB.
> The volume itself only reports to be 26 GB.
>
> Anybody know what is going on here? Is this a JR or fuse-davfs
> configuration? Is there some magic to mounting it? Searched the list but
> haven't identified anything that answers my questions.
>
> Thanks,
> Bill OConnor
>
> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> This email is confidential to the intended recipient. If you have received
> it in error, please notify the sender and delete it from your
> system. Any unauthorized use, disclosure or copying is not permitted. The
> views or opinions presented are solely those of the sender and do
> not necessarily represent those of Public Library of Science unless
> otherwise specifically stated. Please note that neither Public Library
> of Science nor any of its agents accept any responsibility for any viruses
> that may be contained in this e-mail or its attachments and it
> is your responsibility to scan the e-mail and attachments (if any).
>