You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Steve Williams <st...@kromestudios.com> on 2003/07/21 04:27:56 UTC
Corrupt repository on large commit (was Re: svn+ssh on Win32)
>> As for the checkout -- "chunk read error"? Never even heard of that
>> error... from anyone, actually, ever. I'd like to hear more about
>> it.
>
> I'll copy and paste the exact text of the error when I get to the
> office tomorrow.
After checking out approximately half of the repository, I got this...
svn: RA layer request failed
svn: REPORT request failed on '/repos/ty2/!svn/vcc/default'
svn: REPORT of '/repos/ty2/!svn/vcc/default': Error reading chunked response
body (http://julius)
I did a recover on the repository, tried the checkout again and got the same
result.
The directory tree was 2.07GB of 15,793 files in 102 directories. I had
committed the majority of the tree in small sections (very time consuming)
but it was towards the end when one directory tree of 285MB in 9,761 files
in 44 directories failed with a MERGE timeout. This resulted in the
corrupted repository described above.
No mod_deflate in httpd. No mention of 'deflate' in httpd.conf.
http-compression set to "no" in global section of
%APPDATA%\Subversion\servers
Add and commit a large (2.07GB, 15,793 files, 102 directories) directory
tree.
I am trying again with a fresh repository.
The Data directory contains all 2.07GB of files to be added to the
repository.
F:\test>svn add Data
F:\test>svn commit Data -m "Initial commit"
After almost two hours of watching dots go past, it got to a point where the
dots stopped, then there was a huge amount of network traffic followed by a
huge amount of drive access on the client side. So much drive activity that
it slowed the client machine down to a crawl. It was after several minutes
of this client side drive activity that the MERGE timeout occurred.
....................svn: RA layer request failed
svn: Commit failed (details follow):
svn: MERGE request failed on '/repos/test/trunk/data'
svn: MERGE of '/repos/test/trunk/data': timed out waiting for server
(http://julius)
Delete F:\test and try a checkout.
F:\>svn co http://julius/repos/test/trunk test
Checks out thousands of files and then comes up with...
svn: RA layer request failed
svn: REPORT request failed on '/repos/test/!svn/vcc/default'
svn: REPORT of '/repos/test/!svn/vcc/default': Error reading chunked
response body (http://julius)
Trying the same thing with svn: protocol now and it seems to have hit a wall
at a few thousand files into the commit. There is very little disk activity
on the client side, constant disk activity on the server side (the HDD light
is full on and I'm the only person using it) and regular network activity,
but there has been no more output from the client. It's been like that for
almost an hour now. I don't know what it is doing. I cannot communicate
with the server. An existing ssh session is responding extremely slowly
(one character echoed every ten to twenty seconds, if that). The P3-733
server has been brought to its knees. Just wondering if it has got itself
into an endless loop of sorts?
Sly
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupt repository on large commit (was Re: svn+ssh on Win32)
Posted by Steve Williams <st...@kromestudios.com>.
There's about 40GB free on the server. So unless committing 2GB uses over
40GB of drive space, I would guess not. The svnserve process was hammering
the server drive so hard that all other processes on the machine ground to a
near halt. Other than the generic 'Write failure' error message returned by
the svn client, I have no idea what went wrong.
Sly
> > svn: An existing connection was forcibly closed by the remote host.
> > svn: Commit failed (details follow):
> > svn: Write failure
>
> What, did the disk fill up or something?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupt repository on large commit (was Re: svn+ssh on Win32)
Posted by Ben Collins-Sussman <su...@collab.net>.
"Steve Williams" <st...@kromestudios.com> writes:
> svn: An existing connection was forcibly closed by the remote host.
> svn: Commit failed (details follow):
> svn: Write failure
What, did the disk fill up or something?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupt repository on large commit (was Re: svn+ssh on Win32)
Posted by Steve Williams <st...@kromestudios.com>.
> Trying the same thing with svn: protocol now and it seems to have hit
> a wall at a few thousand files into the commit. There is very little
> disk activity on the client side, constant disk activity on the
> server side (the HDD light is full on and I'm the only person using
> it) and regular network activity, but there has been no more output
> from the client. It's been like that for almost an hour now. I
> don't know what it is doing. I cannot communicate with the server.
> An existing ssh session is responding extremely slowly (one character
> echoed every ten to twenty seconds, if that). The P3-733 server has
> been brought to its knees. Just wondering if it has got itself into
> an endless loop of sorts?
It finally stopped after another hour with
svn: An existing connection was forcibly closed by the remote host.
svn: Commit failed (details follow):
svn: Write failure
Sly
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupt repository on large commit (was Re: svn+ssh on Win32)
Posted by Ben Collins-Sussman <su...@collab.net>.
"Steve Williams" <st...@kromestudios.com> writes:
> > > Project #1 takes much longer (longer than Project #2) to perform the
> > > initial checkout.
> > > Again, it is my belief that it is the number of files, not the size of
> > > them, that causes the checkout to take so long.
> >
> > Yes, absolutely. A huge number of files will cause a huge REPORT
> > response from the server. If the server is *buffering* the entire
> > report before sending it to you, it will take a *really* long time
> > before you see a single file added to your working copy.
> >
> > Please please please, upgrade to 0.25, and turn off mod_deflate.
>
> As I said in my response, the client was hammering the local drive when the
> timeout error occurred. There was no network activity and no server drive
> activity. Surely this is not a sign that the server is buffering the
> report? I think the client was dealing with the report at the time the
> timeout error occurred.
Which is amazing to me... how can neon "timeout" on a response, when
it's actually *parsing* the response. I believe you when you said
that no compression was being used.
Your analysis is correct -- the client disk-thrash you hear is ra_dav
parsing the huge server response, and updating metadata in your
working copy about the results of the commit.
Steve, this is now filed as issue #1430. Can you add yourself to the
cc: list of that issue?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupt repository on large commit (was Re: svn+ssh on Win32)
Posted by Steve Williams <st...@kromestudios.com>.
> > Project #1 takes much longer (longer than Project #2) to perform the
> > initial checkout.
> > Again, it is my belief that it is the number of files, not the size of
> > them, that causes the checkout to take so long.
>
> Yes, absolutely. A huge number of files will cause a huge REPORT
> response from the server. If the server is *buffering* the entire
> report before sending it to you, it will take a *really* long time
> before you see a single file added to your working copy.
>
> Please please please, upgrade to 0.25, and turn off mod_deflate.
As I said in my response, the client was hammering the local drive when the
timeout error occurred. There was no network activity and no server drive
activity. Surely this is not a sign that the server is buffering the
report? I think the client was dealing with the report at the time the
timeout error occurred.
Sly
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupt repository on large commit (was Re: svn+ssh on Win32)
Posted by Ben Collins-Sussman <su...@collab.net>.
Russell Glaue <rg...@cait.org> writes:
> number corrections.
> Project #1: 13,000 files, 789 MB
> Project #2: 5,000 files, 892 MB
>
> Project #1 takes much longer (longer than Project #2) to perform the
> initial checkout.
> Again, it is my belief that it is the number of files, not the size of
> them, that causes the checkout to take so long.
Yes, absolutely. A huge number of files will cause a huge REPORT
response from the server. If the server is *buffering* the entire
report before sending it to you, it will take a *really* long time
before you see a single file added to your working copy.
Please please please, upgrade to 0.25, and turn off mod_deflate.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupt repository on large commit (was Re: svn+ssh on Win32)
Posted by Russell Glaue <rg...@cait.org>.
number corrections.
Project #1: 13,000 files, 789 MB
Project #2: 5,000 files, 892 MB
Project #1 takes much longer (longer than Project #2) to perform the
initial checkout.
Again, it is my belief that it is the number of files, not the size of
them, that causes the checkout to take so long.
-RG
On Monday, Jul 21, 2003, at 09:51 America/Chicago, Russell Glaue wrote:
> We got errors similar to this too.
> We are checking in 2000 files or more and it croaks close to the end.
> We ended up removing the repository and doing the check-in again
> (crossing our fingers).
> One repos with well over 3000 files always croaked and we had to split
> the check-in to 3 phases.
> A second repository was about 2000 files and it croaked the first time
> and second time. The third time it made it all the way through without
> croaking on us.
> This was with Subversion server 0.17 though.
>
> If you have 0.25, try the suggestions discussed here (upgrade to 0.25
> if you do not have it).
> set local config 'http-compression = no'
> set local config 'http-timeout = 7200'
>
> We have only did the later, and things seem a little better. We will
> try turning compression off if problems arise again.
>
> For those who are interested in the time for check-in. The above two
> repositories took about 2 hours plus to complete the check-in. This is
> only what I remember.
> -RG
>
>
> On Sunday, Jul 20, 2003, at 23:27 America/Chicago, Steve Williams
> wrote:
>
>>>> As for the checkout -- "chunk read error"? Never even heard of that
>>>> error... from anyone, actually, ever. I'd like to hear more about
>>>> it.
>>>
>>> I'll copy and paste the exact text of the error when I get to the
>>> office tomorrow.
>>
>> After checking out approximately half of the repository, I got this...
>>
>> svn: RA layer request failed
>> svn: REPORT request failed on '/repos/ty2/!svn/vcc/default'
>> svn: REPORT of '/repos/ty2/!svn/vcc/default': Error reading chunked
>> response
>> body (http://julius)
>>
>> I did a recover on the repository, tried the checkout again and got
>> the same
>> result.
>>
>> The directory tree was 2.07GB of 15,793 files in 102 directories. I
>> had
>> committed the majority of the tree in small sections (very time
>> consuming)
>> but it was towards the end when one directory tree of 285MB in 9,761
>> files
>> in 44 directories failed with a MERGE timeout. This resulted in the
>> corrupted repository described above.
>>
>> No mod_deflate in httpd. No mention of 'deflate' in httpd.conf.
>> http-compression set to "no" in global section of
>> %APPDATA%\Subversion\servers
>> Add and commit a large (2.07GB, 15,793 files, 102 directories)
>> directory
>> tree.
>>
>> I am trying again with a fresh repository.
>>
>> The Data directory contains all 2.07GB of files to be added to the
>> repository.
>>
>> F:\test>svn add Data
>> F:\test>svn commit Data -m "Initial commit"
>>
>> After almost two hours of watching dots go past, it got to a point
>> where the
>> dots stopped, then there was a huge amount of network traffic
>> followed by a
>> huge amount of drive access on the client side. So much drive
>> activity that
>> it slowed the client machine down to a crawl. It was after several
>> minutes
>> of this client side drive activity that the MERGE timeout occurred.
>>
>> ....................svn: RA layer request failed
>> svn: Commit failed (details follow):
>> svn: MERGE request failed on '/repos/test/trunk/data'
>> svn: MERGE of '/repos/test/trunk/data': timed out waiting for server
>> (http://julius)
>>
>> Delete F:\test and try a checkout.
>>
>> F:\>svn co http://julius/repos/test/trunk test
>>
>> Checks out thousands of files and then comes up with...
>>
>> svn: RA layer request failed
>> svn: REPORT request failed on '/repos/test/!svn/vcc/default'
>> svn: REPORT of '/repos/test/!svn/vcc/default': Error reading chunked
>> response body (http://julius)
>>
>> Trying the same thing with svn: protocol now and it seems to have hit
>> a wall
>> at a few thousand files into the commit. There is very little disk
>> activity
>> on the client side, constant disk activity on the server side (the
>> HDD light
>> is full on and I'm the only person using it) and regular network
>> activity,
>> but there has been no more output from the client. It's been like
>> that for
>> almost an hour now. I don't know what it is doing. I cannot
>> communicate
>> with the server. An existing ssh session is responding extremely
>> slowly
>> (one character echoed every ten to twenty seconds, if that). The
>> P3-733
>> server has been brought to its knees. Just wondering if it has got
>> itself
>> into an endless loop of sorts?
>>
>> Sly
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
>> For additional commands, e-mail: dev-help@subversion.tigris.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: Corrupt repository on large commit (was Re: svn+ssh on Win32)
Posted by Russell Glaue <rg...@cait.org>.
We got errors similar to this too.
We are checking in 2000 files or more and it croaks close to the end.
We ended up removing the repository and doing the check-in again
(crossing our fingers).
One repos with well over 3000 files always croaked and we had to split
the check-in to 3 phases.
A second repository was about 2000 files and it croaked the first time
and second time. The third time it made it all the way through without
croaking on us.
This was with Subversion server 0.17 though.
If you have 0.25, try the suggestions discussed here (upgrade to 0.25
if you do not have it).
set local config 'http-compression = no'
set local config 'http-timeout = 7200'
We have only did the later, and things seem a little better. We will
try turning compression off if problems arise again.
For those who are interested in the time for check-in. The above two
repositories took about 2 hours plus to complete the check-in. This is
only what I remember.
-RG
On Sunday, Jul 20, 2003, at 23:27 America/Chicago, Steve Williams wrote:
>>> As for the checkout -- "chunk read error"? Never even heard of that
>>> error... from anyone, actually, ever. I'd like to hear more about
>>> it.
>>
>> I'll copy and paste the exact text of the error when I get to the
>> office tomorrow.
>
> After checking out approximately half of the repository, I got this...
>
> svn: RA layer request failed
> svn: REPORT request failed on '/repos/ty2/!svn/vcc/default'
> svn: REPORT of '/repos/ty2/!svn/vcc/default': Error reading chunked
> response
> body (http://julius)
>
> I did a recover on the repository, tried the checkout again and got
> the same
> result.
>
> The directory tree was 2.07GB of 15,793 files in 102 directories. I
> had
> committed the majority of the tree in small sections (very time
> consuming)
> but it was towards the end when one directory tree of 285MB in 9,761
> files
> in 44 directories failed with a MERGE timeout. This resulted in the
> corrupted repository described above.
>
> No mod_deflate in httpd. No mention of 'deflate' in httpd.conf.
> http-compression set to "no" in global section of
> %APPDATA%\Subversion\servers
> Add and commit a large (2.07GB, 15,793 files, 102 directories)
> directory
> tree.
>
> I am trying again with a fresh repository.
>
> The Data directory contains all 2.07GB of files to be added to the
> repository.
>
> F:\test>svn add Data
> F:\test>svn commit Data -m "Initial commit"
>
> After almost two hours of watching dots go past, it got to a point
> where the
> dots stopped, then there was a huge amount of network traffic followed
> by a
> huge amount of drive access on the client side. So much drive
> activity that
> it slowed the client machine down to a crawl. It was after several
> minutes
> of this client side drive activity that the MERGE timeout occurred.
>
> ....................svn: RA layer request failed
> svn: Commit failed (details follow):
> svn: MERGE request failed on '/repos/test/trunk/data'
> svn: MERGE of '/repos/test/trunk/data': timed out waiting for server
> (http://julius)
>
> Delete F:\test and try a checkout.
>
> F:\>svn co http://julius/repos/test/trunk test
>
> Checks out thousands of files and then comes up with...
>
> svn: RA layer request failed
> svn: REPORT request failed on '/repos/test/!svn/vcc/default'
> svn: REPORT of '/repos/test/!svn/vcc/default': Error reading chunked
> response body (http://julius)
>
> Trying the same thing with svn: protocol now and it seems to have hit
> a wall
> at a few thousand files into the commit. There is very little disk
> activity
> on the client side, constant disk activity on the server side (the HDD
> light
> is full on and I'm the only person using it) and regular network
> activity,
> but there has been no more output from the client. It's been like
> that for
> almost an hour now. I don't know what it is doing. I cannot
> communicate
> with the server. An existing ssh session is responding extremely
> slowly
> (one character echoed every ten to twenty seconds, if that). The
> P3-733
> server has been brought to its knees. Just wondering if it has got
> itself
> into an endless loop of sorts?
>
> Sly
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org