E175012 and E175002 when trying to commit

[Christoph Vogtländer <Ch...@sigma-surface-science.com>]

Hi list,

committing a huge change-set (about 110.000 new files, 10.000 deletions, 50.000 modified file, about 2GiB data payload) fails with a timeout during transaction submit:

$> svn ci -F svn-commit.tmp .
Sende ...
Lösche ...
Füge hinzu ...
...
Übertrage Daten ........
...
.................erledigt
Übertrage Transaktion...
svn: E175012: Übertragen schlug fehl (Details folgen):
svn: E175012: Die Wartezeit für die Verbindung ist abgelaufen
svn: E200042: Zusätzliche Fehler:
svn: E175012: Die Wartezeit für die Verbindung ist abgelaufen



Before posting this to the mailing list I wanted to get the message in English. But, changing LANG also changed the behaviour.
The client no longer tries to submit the transaction at all but fails straight away with:

$> LANG=en_GB.UTF-8 svn ci -F svn-commit.tmp .
svn: E175002: Commit failed (details follow):
svn: E175002: Unexpected HTTP status 413 'Request Entity Too Large' on '/svn/ThirdParty/!svn/me'

This is reproducible. Running with de_DE.UTF-8 (my default LANG), svn will try to submit the change but runs into a time-out. Running with en_GB.UTF-8, svn fails with error 413.

Why is subversion acting differently with different language setting on the client side?

Any ideas what can be done to successfully commit the changes?

Client
svn --version
svn, Version 1.11.1 (r1850623)

Server:
VisualSVN Server 3.9.2 (Subversion 1.10.3, Apache 2.4.35)

--
Regards
Christoph


Sigma Surface Science GmbH, Idsteiner Str. 78, 65232 Taunusstein. Amtsgericht Wiesbaden, HRB 27422. Geschäftsführer: Norbert Nold

AW: E175012 and E175002 when trying to commit

[Christoph Vogtländer <Ch...@sigma-surface-science.com>]

Hi Pavel,

> Von: Pavel Lyalyakin [mailto:pavel.lyalyakin@visualsvn.com]
> Gesendet: Donnerstag, 4. April 2019 15:18
>
> I see that you use VisualSVN Server. But do you use VDFS (Multisite Repository Replication)? Do you see the following error when you commit to a slave VDFS repository?
> [[[
> svn: E175012: Commit failed (details follow):
> svn: E175012: Connection timed out
> svn: E160032: Additional errors:
> svn: E160032: Cannot abort transaction '84-2m', because it is still being processed.
> ]]]

No, we are not using VDFS. The repo is FSFS with filesystem format "8" (with lz4 compression)


Regards
Christoph
Sigma Surface Science GmbH, Idsteiner Str. 78, 65232 Taunusstein. Amtsgericht Wiesbaden, HRB 27422. Geschäftsführer: Norbert Nold

Re: E175012 and E175002 when trying to commit

[Pavel Lyalyakin <pa...@visualsvn.com>]

Hello Christoph,

On Thu, Apr 4, 2019 at 12:39 PM Christoph Vogtländer <
Christoph.Vogtlaender@sigma-surface-science.com> wrote:

> I have now a working setup with
>
> LimitXMLRequestBody 32000000
> Timeout 36000
>
> I can successfully commit the change-set using LANG=de_DE.UTF-8,
> LANG=en_GB.UTF-8 or LC_TYPE=C
>
> To wrap this up:
> It seems there is a bug when using de_DE.UTF-8.
> The commit should be denied in case the XML request body is too large,
> which is not the case.
> During transmit, the server will cancel with error.
> As also the timeout was reached, the client reports only a timeout.
>
> When enlarging the LimitXMLRequestBody, only, the commit will be
> transmitted successfully. But,
> the client will still get a timeout (as the commit will last a lot longer
> than the 300s configured by
> default). Also enlarging the timeout parameter value will fix this.
>
> So, beside the failing test for the XML request body size with de_DE.UTF-8
> everything is working
> as designed :)
>

I see that you use VisualSVN Server. But do you use VDFS (Multisite
Repository Replication)? Do you see the following error when you commit to
a slave VDFS repository?
[[[
svn: E175012: Commit failed (details follow):
svn: E175012: Connection timed out
svn: E160032: Additional errors:
svn: E160032: Cannot abort transaction '84-2m', because it is still being
processed.
]]]

If that's your case, what is the replications link's bandwidth?

-- 
With best regards,
Pavel Lyalyakin
VisualSVN Team

AW: E175012 and E175002 when trying to commit

[Christoph Vogtländer <Ch...@sigma-surface-science.com>]

I have now a working setup with

LimitXMLRequestBody 32000000
Timeout 36000

I can successfully commit the change-set using LANG=de_DE.UTF-8, LANG=en_GB.UTF-8 or LC_TYPE=C

To wrap this up:
It seems there is a bug when using de_DE.UTF-8.
The commit should be denied in case the XML request body is too large, which is not the case.
During transmit, the server will cancel with error.
As also the timeout was reached, the client reports only a timeout.

When enlarging the LimitXMLRequestBody, only, the commit will be transmitted successfully. But,
the client will still get a timeout (as the commit will last a lot longer than the 300s configured by
default). Also enlarging the timeout parameter value will fix this.

So, beside the failing test for the XML request body size with de_DE.UTF-8 everything is working
as designed :)

Regards
Christoph

Sigma Surface Science GmbH, Idsteiner Str. 78, 65232 Taunusstein. Amtsgericht Wiesbaden, HRB 27422. Geschäftsführer: Norbert Nold

AW: E175012 and E175002 when trying to commit

[Christoph Vogtländer <Ch...@sigma-surface-science.com>]

> Von: Daniel Shahaf [mailto:d.s@daniel.shahaf.name]
> Gesendet: Dienstag, 2. April 2019 22:20
> I'm not sure, especially given that both locales are UTF-8 ones.  A few
> shots in the dark:
>
> - Try '-F /dev/null' instead of '-F svn-commit.tmp'.

This will fail with E175012 (but with different "details"):
svn: E175012: Übertragen schlug fehl (Details folgen):
svn: E175012: Die Wartezeit für die Verbindung ist abgelaufen
svn: E200042: Zusätzliche Fehler:
svn: E175002: Unerwarteter Serverfehler 500 »Internal Server Error« auf »/svn/ThirdParty/!svn/txn/914-pj«

On the server I get:
Could not open specified transaction.  [500, #160014]
Reference to non-existent node '0.0.t914-pj' in filesystem 'D:/Repositories/ThirdParty/db'  [500, #160014]


> - Is there any non-ASCII in the output of `svn st -q`?  (`svn st -q | sed 's/[ -
> ~]//g' | grep .`; that's left bracket, space, minus, tilde, right bracket)

This exists with exit code 1. So I assume there are no non-ASCII character?

> - Try LC_ALL=C and LC_ALL=C.UTF-8.

$> LC_ALL=C svn ci -F svn-commit.tmp .
svn: E175002: Commit failed (details follow):
svn: E175002: Unexpected HTTP status 413 'Request Entity Too Large' on '/svn/ThirdParty/!svn/me'

$> LC_ALL=C.UTF-8 svn ci -F svn-commit.tmp .
svn: warning: cannot set LC_CTYPE locale
svn: warning: environment variable LC_ALL is C.UTF-8
svn: warning: please check that your locale name is correct
svn: E175002: Commit failed (details follow):
svn: E175002: Unexpected HTTP status 413 'Request Entity Too Large' on '/svn/ThirdParty/!svn/me'


On the server I get:
Request body is larger than the configured limit of 8000000
Error parsing skel POST request body.  [413, #175002]

> > Any ideas what can be done to successfully commit the changes?
>
> Depends.  If you want to make the changes in one commit, you'll need to
> increase the timeouts in server and client sufficiently (that includes
> any proxies).  A workaround is to separate the changes to several
> commits: use 'svn commit -- ./foo ./bar' instead of 'svn commit -- ./'.

I'd rather want to commit in a single commit. I also need to merge this change into trunk and
commit there as well. Most likely I will run into this issue again, later.

The "8000000" seems to refer to the "LimitXMLRequestBody" property in httpd.conf.
I increased the " LimitXMLRequestBody" to 32000000 in httpd-custom.conf. The transaction
was committed successfully using LC_ALL=C. But, the client still exited with a time-out.
I solved this by reverting and updating the working copy. I will try to increase the parameter
"Timeout" as well to see if this makes any difference for the client.
I will also try to use de_DE.UTF-8 after merging into to trunk to see if this is now working as well.


So it seems the behaviour using local C or en_GB.UTF-8 is working correctly (checking
the request body size before starting the commit) whereas using de_DE.UTF-8 does not
seem to check correctly resulting in a transaction failure.

Sigma Surface Science GmbH, Idsteiner Str. 78, 65232 Taunusstein. Amtsgericht Wiesbaden, HRB 27422. Geschäftsführer: Norbert Nold

AW: E175012 and E175002 when trying to commit

[Christoph Vogtländer <Ch...@sigma-surface-science.com>]

> Von: Johan Corveleyn [mailto:jcorvel@gmail.com]
> Gesendet: Mittwoch, 3. April 2019 10:18
> Perhaps the difference between your first and second attempt is not
> caused by different language settings, but because the working copy
> was in a different state? For instance, if the "base" of your working
> copy is full of mixed revisions, and you're committing a lot of
> changes on top of that, the "edit report" that's sent by the client to
> the server can be much larger (thus hitting some limit on the server).
> Maybe try to execute another 'svn update' on your entire working copy
> so it gets a uniform revision.

No, I don't think so. I'm using a fresh check-out on the branch, there
should be no mixed revisions.
And this issue is reproducible. Running with de_DE.UTF-8 will always try to
send the data, using en_GB.UTF-8 will fail straight away with
'Request Entity Too Large' error. I run these commands in the same local
working copy. Nothing else in between (like cleanup etc.)

> Just another hint to get you started: the client-side timeout can be
> found in the file ~/.subversion/servers on your client machine. Look
> for "http-timeout = ..." (the default is 60 (seconds) I believe).

Nothing set on the client side. The server defines a timeout of 300. When using
de_DE.UTF-8 the commit fails after over 90 Minutes. So this settings does not
seem to be related. But, I will try to increase the timeout and check if that helps.

> Though I would seriously recommend splitting up your huge commit into
> several commits. Perhaps find some logical blocks, or split by
> subdirectories, or ... Huge commits can be a pain later on, for
> instance because every time you query the 'svn log' for one of those
> files later, this huge commit will be part of the result (part of its
> history), and with the '-v' (verbose) option 'svn log' will show all
> the affected paths, which will be huge ...

Unfortunately, this is an import from 3rd party. So, there are no logical blocks.
I might be able to commit on sub-folder after the other. But, I need to merge
this into trunk later which will then be a lot of hassle and error prone.

I will try to commit in a single commit after adapting the server configuration
(Timeout, LimitXMLRequestBody), first.


Sigma Surface Science GmbH, Idsteiner Str. 78, 65232 Taunusstein. Amtsgericht Wiesbaden, HRB 27422. Geschäftsführer: Norbert Nold

Re: E175012 and E175002 when trying to commit

[Johan Corveleyn <jc...@gmail.com>]

On Tue, Apr 2, 2019 at 10:20 PM Daniel Shahaf <d....@daniel.shahaf.name> wrote:
>
> Christoph Vogtländer wrote on Tue, 02 Apr 2019 09:02 +00:00:
> > This is reproducible. Running with de_DE.UTF-8 (my default LANG), svn
> > will try to submit the change but runs into a time-out. Running with
> > en_GB.UTF-8, svn fails with error 413.
> >
> > Why is subversion acting differently with different language setting on
> > the client side?
> >
>
> I'm not sure, especially given that both locales are UTF-8 ones.

Perhaps the difference between your first and second attempt is not
caused by different language settings, but because the working copy
was in a different state? For instance, if the "base" of your working
copy is full of mixed revisions, and you're committing a lot of
changes on top of that, the "edit report" that's sent by the client to
the server can be much larger (thus hitting some limit on the server).
Maybe try to execute another 'svn update' on your entire working copy
so it gets a uniform revision.

> A few
> shots in the dark:
>
> - Try '-F /dev/null' instead of '-F svn-commit.tmp'.
> - Is there any non-ASCII in the output of `svn st -q`?  (`svn st -q | sed 's/[ -~]//g' | grep .`; that's left bracket, space, minus, tilde, right bracket)
> - Try LC_ALL=C and LC_ALL=C.UTF-8.
>
> > Any ideas what can be done to successfully commit the changes?
>
> Depends.  If you want to make the changes in one commit, you'll need to
> increase the timeouts in server and client sufficiently (that includes
> any proxies).  A workaround is to separate the changes to several
> commits: use 'svn commit -- ./foo ./bar' instead of 'svn commit -- ./'.

Just another hint to get you started: the client-side timeout can be
found in the file ~/.subversion/servers on your client machine. Look
for "http-timeout = ..." (the default is 60 (seconds) I believe).

Also, on the server-side, the httpd directive LimitXMLRequestBody
might be playing a role in that second error ("Request Entity Too
Large"). In our setup at work we have "LimitXMLRequestBody 0" in our
httpd config, to effectively have no limit (for a publicly exposed
repository you should always have some limit, to avoid malicious users
being able to crash your server, but if you only have internal users I
think it's OK to eliminate that limit).

Though I would seriously recommend splitting up your huge commit into
several commits. Perhaps find some logical blocks, or split by
subdirectories, or ... Huge commits can be a pain later on, for
instance because every time you query the 'svn log' for one of those
files later, this huge commit will be part of the result (part of its
history), and with the '-v' (verbose) option 'svn log' will show all
the affected paths, which will be huge ...

> I'm not sure why you'd get a 413 on the request to !svn/me, though.  (to
> the list) Are any requests made to !svn/me other than the initial POST?

Sorry, no idea.

-- 
Johan

Re: E175012 and E175002 when trying to commit

[Daniel Shahaf <d....@daniel.shahaf.name>]

Christoph Vogtländer wrote on Tue, 02 Apr 2019 09:02 +00:00:
> This is reproducible. Running with de_DE.UTF-8 (my default LANG), svn 
> will try to submit the change but runs into a time-out. Running with 
> en_GB.UTF-8, svn fails with error 413.
> 
> Why is subversion acting differently with different language setting on 
> the client side?
> 

I'm not sure, especially given that both locales are UTF-8 ones.  A few
shots in the dark:

- Try '-F /dev/null' instead of '-F svn-commit.tmp'.
- Is there any non-ASCII in the output of `svn st -q`?  (`svn st -q | sed 's/[ -~]//g' | grep .`; that's left bracket, space, minus, tilde, right bracket)
- Try LC_ALL=C and LC_ALL=C.UTF-8.

> Any ideas what can be done to successfully commit the changes?

Depends.  If you want to make the changes in one commit, you'll need to
increase the timeouts in server and client sufficiently (that includes
any proxies).  A workaround is to separate the changes to several
commits: use 'svn commit -- ./foo ./bar' instead of 'svn commit -- ./'.

I'm not sure why you'd get a 413 on the request to !svn/me, though.  (to
the list) Are any requests made to !svn/me other than the initial POST?