You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Davor Josipovic <da...@live.com> on 2018/05/21 20:05:05 UTC
Re: Reference to non-existent node
I have finally found the culprit. By coincidence, I had the same configuration running on two servers, one backed with a SSD, the other with a HDD.
The same commit worked on the SSD-backed server, and always failed on the HDD one with the described error.
Monitoring the process revealed that after the file transfer of the commit completes, server starts finalizing. Not sure what exactly happens during that period serverside (maybe someone can explain?), but for a 40k files commit, it took the SSD-backed one to complete in about 30 seconds, while the HDD-backed one struggled more than 30 minutes. During those 30 minutes, the client got no response, so it timed out, which resulted in the described error on the server side.
Anyway, setting http-timeout = 3600 in the "servers" file fixes the issue permanently (tested it 2 times to make sure). I think the default is 10 minutes.
Also, I am quite surprised at the performance penalty when using a HDD at this stage. Any way to tweak this for low memory servers?
On 21 February 2018 at 16:27, Philip Martin <ph...@codematters.co.uk>> wrote:
Davor Josipovic <da...@live.com>> writes:
> You make the PID sound very important, and with good reason too! I
> just went back through the logs to make sure that I didn't mess up the
> PID before posting. I didn't. In each case, the PID was
> different. This might explain the time difference.
The Subversion client can make multiple http connections, particularly
when a commit involves a large number of files. It is not unusual for
multiple PIDs to handle the requests for any one commit.
> I'll look further into this later. The default Debian apache2 install
> came with mpm_event enabled. According to my logs I switched to
> mpm_prefork later (due to performance reasons recommended for an other
> module).
Prefork is typically the lowest performace for Subversion. Subversion
server performance will generally be better with a small number of
processes each with a large number of threads as this allows you to
increase the size of the per-process Subversion cache.
--
Philip
Re: Reference to non-existent node
Posted by Davor Josipovic <da...@live.com>.
> Now that is interesting. 40k doesn't seem to be such a large amount of data for modern computers. Very slow and fragmented hard drive? Or perhaps there's something else going on that is manifesting this way?
The HDD is indeed on the slowside, and together with low memory...
But I think this also show how I/O intensive SVN is. On the client side, for each committed file, one copy is placed in .svn folder, and an other copy in a temporary folder (which is deleted after file transfer in v1.9). So for each file committed, a double copy is made client-side. This temporary copy is really necessary?
Server-side, I see similar disk bashing. For each committed file, max 2 (?) copies are made in the transaction directory.
So any way to reduce the I/O?
Re: Reference to non-existent node
Posted by Johan Corveleyn <jc...@gmail.com>.
On Tue, May 22, 2018 at 8:30 PM, Davor Josipovic <da...@live.com> wrote:
>> Now that is interesting. 40k doesn't seem to be such a large amount of
>> data for modern computers. Very slow and fragmented hard drive? Or perhaps
>> there's something else going on that is manifesting this way?
>
> The HDD is indeed on the slowside, and together with low memory...
>
> But I think this also show how I/O intensive SVN is. On the client side, for
> each committed file, one copy is placed in .svn folder, and an other copy in
> a temporary folder (which is deleted after file transfer in v1.9). So for
> each file committed, a double copy is made client-side. This temporary copy
> is really necessary?
>
> Server-side, I see similar disk bashing. For each committed file, max 2 (?)
> copies are made in the transaction directory.
>
> So any way to reduce the I/O?
A couple of improvements have been made in 1.10. It would be
interesting to test this same scenario with a 1.10 client (to verify
the client-side improvements) and if possible also with a 1.10 server,
with a dumped+loaded FSFS back-end (to get an idea of any server-side
improvements).
Client-side there were some changes to make the (http) client stream
svndiff deltas without creating temporary files (during 'commit' and
'import'):
https://svn.apache.org/r1803143
https://svn.apache.org/r1803144
https://svn.apache.org/r1804819
Server-side, LZ4 compression was added as the default compression
algorithm in the FSFS back-end, and on the wire (not sure if that
helps in your case ... it won't impact the number of files that are
created temporarily I suppose):
https://subversion.apache.org/docs/release-notes/1.10.html#lz4
Also, maybe a commit such as this one helps too (though it seems to
only talk about reads):
https://svn.apache.org/r1759399
As always, probably further improvements can be made I guess, but
someone will need to dig into it.
If you'd be able to test with 1.10 both client and server-side, that
would give some more data points about where we are with the latest
state of things ...
Finally, one last tip: this (rather old) performance tuning page
suggests putting the "transactions" folder on a separate (faster)
filesystem to speed up commits:
https://www.orcaware.com/svn/wiki/Server_performance_tuning_for_Linux_and_Unix#Reducing_the_number_of_writes
I don't know how realistic that is in your setup (plus you need to
make sure that transactions filesystem is large enough for your
commits).
--
Johan
Re: Reference to non-existent node
Posted by Davor Josipovic <da...@live.com>.
> Now that is interesting. 40k doesn't seem to be such a large amount of data for modern computers. Very slow and fragmented hard drive? Or perhaps there's something else going on that is manifesting this way?
The HDD is indeed on the slowside, and together with low memory...
But I think this also show how I/O intensive SVN is. On the client side, for each committed file, one copy is placed in .svn folder, and an other copy in a temporary folder (which is deleted after file transfer in v1.9). So for each file committed, a double copy is made client-side. This temporary copy is really necessary?
Server-side, I see similar disk bashing. For each committed file, max 2 (?) copies are made in the transaction directory.
So any way to reduce the I/O?
Re: Reference to non-existent node
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Tue, May 22, 2018 at 2:36 PM Davor Josipovic <da...@live.com> wrote:
> > Now that is interesting. 40k doesn't seem to be such a large amount of
data for modern computers. Very slow and fragmented hard drive? Or perhaps
there's something else going on that is manifesting this way?
> The HDD is indeed on the slowside, and together with low memory...
> But I think this also show how I/O intensive SVN is. On the client side,
for each committed file, one copy is placed in .svn folder, and an other
copy in a temporary folder (which is deleted after file transfer in v1.9).
So for each file committed, a double copy is made client-side. This
temporary copy is really necessary?
I think it shows how I/O intensive using 40,000 small files is. Especially
if they are in the same directory, many filesystems get increasingly
unhappy as they try to manage that many files in one directory.
Re: Reference to non-existent node
Posted by Davor Josipovic <da...@live.com>.
> Now that is interesting. 40k doesn't seem to be such a large amount of data for modern computers. Very slow and fragmented hard drive? Or perhaps there's something else going on that is manifesting this way?
The HDD is indeed on the slowside, and together with low memory...
But I think this also show how I/O intensive SVN is. On the client side, for each committed file, one copy is placed in .svn folder, and an other copy in a temporary folder (which is deleted after file transfer in v1.9). So for each file committed, a double copy is made client-side. This temporary copy is really necessary?
Server-side, I see similar disk bashing. For each committed file, max 2 (?) copies are made in the transaction directory.
So any way to reduce the I/O?
Re: Reference to non-existent node
Posted by Nathan Hartman <ha...@gmail.com>.
On Mon, May 21, 2018 at 4:05 PM Davor Josipovic <da...@live.com> wrote:
> I have finally found the culprit. By coincidence, I had the same
> configuration running on two servers, one backed with a SSD, the other with
> a HDD.
>
> The same commit worked on the SSD-backed server, and always failed on the
> HDD one with the described error.
>
> Monitoring the process revealed that after the file transfer of the commit
> completes, server starts finalizing. Not sure what exactly happens during
> that period serverside (maybe someone can explain?), but for a 40k files
> commit, it took the SSD-backed one to complete in about 30 seconds, while
> the HDD-backed one struggled more than 30 minutes. During those 30 minutes,
> the client got no response, so it timed out, which resulted in the
> described error on the server side.
>
(Snip)
Now that is interesting. 40k doesn't seem to be such a large amount of data
for modern computers. Very slow and fragmented hard drive? Or perhaps
there's something else going on that is manifesting this way?