You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Sebastian Tusk <se...@gmx.net> on 2006/03/07 12:46:46 UTC
export, checkout, commit performance
I did some tests with a repository that contains a single file sized 686
MB. I used the svn client version 1.3.0 on a windows machine.
svn commit 1587s
svn export 136s
svn co 395s
To me this figures appeared way to high. It seems that most of the time
is used up in disk activities so I had took a look at that.
svn commit
The file to commit is named test. The function performed is guessed.
activity function? time
------
read 512b blocks from "test"
write 512b blocks to "test.svn_base.tmp" COPY? 3m10s
------
read 102400b blocks from "test.svn-base"
write 4096b blocks to "temp.tmp"
occasionally other reads from "test.svn-base" COPY? DIFF? 2m23s
------
read 512b blocks from test.svn-base HASHING? 1m47s
------
read 4096b blocks from "tempfile" TRANSFER TO SERVER 7m8s
------
read 512b blocks from "test.svn-base" HASHING? 1m16s
------
read 512b blocks from "test"
read 512b blocks from "test.svn-base" COMPARE? 7m46s
------
read 512b blocks from "test.svn-base"
write 512b blocks to "test.tmp.tmp"
occasionally other reads from "test.svn-base" COPY? 3m12s
------
read 512b blocks from "test"
read 512b blocks from "test.tmp" COMPARE? 7m42s
That are all activities that consume significant time. All reads and
writes are over the full 680MB file. Only during TRANSFER TO SERVER
there is any noticible server activity. Alltogether the commit command
uses more than 13 million read or write operations.
Are all this activities necessary? Especially those after the transfer
to the server is finished. Is it necessary to access two files in
parallel in such small blocks? Using larger read/write blocks might
improve the performance significant without to much hassle.
Sebastian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by David Summers <da...@summersoft.fay.ar.us>.
This may be unrelated, but it seems to me that subversion 1.3.0 has
regressed speed-wise (maybe just on Linux, not sure on Windows).
I have a 2 GB working copy that I normally do "svn status", "svn diff",
"svn update", "svn commit", etc. It has a lot of big image files in it as
well as about 200K or more of source code. Up until 1.3.0 I could do
those commands in seconds. Since 1.3.0 it sometimes takes me 30 minutes
or an hour to do some of those operations, but not always, just enough of
the time to make it feel painful. I was doing a commit the other night
that just changed several lines in a file and it took it OVER AN HOUR to
commit. Normally it just takes 5 - 10 seconds.
Not quantifiable evidence, I know, but it sure seems slower.
- David Summers
On Tue, 7 Mar 2006, Julian Foad wrote:
> Sebastian Tusk wrote:
>> I did some tests with a repository that contains a single file sized 686
>> MB. I used the svn client version 1.3.0 on a windows machine.
>>
>> svn commit 1587s
>> svn export 136s
>> svn co 395s
>>
>> To me this figures appeared way to high.
>
> You're right: these figures do appear to be too high, at least for commit and
> checkout. I don't suppose it is that slow on all set-ups, so it might help
> one of us to investigate the problem if you could you give some more details
> about the system:
>
> Subversion client:
> * Subversion v1.3.0
> * running on what version of MS Windows?
> * what filesystem type (FAT, NTFS, etc.)?
>
> Subversion server:
> * which version?
> * which type of repository (FSFS or BDB)?
> * which type of Subversion server (local, svnserve, Apache/DAV)?
> * running on what OS?
> * what filesystem type (FAT, NTFS, etc.)?
>
> Thanks for letting us know of the problem. Maybe someone will be able to
> determine which particular circumstances give the worst performance, and make
> it better.
>
> - Julian
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
--
David Wayne Summers "Linux: Because reboots are for hardware upgrades!"
david@summersoft.fay.ar.us PGP Key: http://summersoft.fay.ar.us/~david/pgp.txt
PGP Key fingerprint = 0B44 B118 85CC F4EC 7021 1ED4 1516 5B78 E320 2001
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Ivan Zhakov <ch...@gmail.com>.
On 3/8/06, Sebastian Tusk <se...@gmx.net> wrote:
> Julian Foad wrote:
> > Sebastian Tusk wrote:
> >
> >> I did some tests with a repository that contains a single file sized
> >> 686 MB. I used the svn client version 1.3.0 on a windows machine.
> >>
> >> svn commit 1587s
> >> svn export 136s
> >> svn co 395s
> >>
> >> To me this figures appeared way to high.
> >
> >
> > You're right: these figures do appear to be too high, at least for
> > commit and checkout. I don't suppose it is that slow on all set-ups, so
> > it might help one of us to investigate the problem if you could you give
> > some more details about the system:
> >
> > Subversion client:
> > * Subversion v1.3.0
> > * running on what version of MS Windows?
> > * what filesystem type (FAT, NTFS, etc.)?
>
> Windows XP Prof
> NTFS
>
> > Subversion server:
> > * which version?
> > * which type of repository (FSFS or BDB)?
> > * which type of Subversion server (local, svnserve, Apache/DAV)?
> > * running on what OS?
> > * what filesystem type (FAT, NTFS, etc.)?
>
> Subversion 1.1.4
> BDB
> Apache/DAV
> Debian 3.1
> ext3
>
> Server and client are connected with 100Mbit Ethernet.
>
> As comparison here the results if I am doing the commit of a 1GB file on
> the server
>
> local through dav
> real 12m5.226s
> user 2m29.560s
> sys 0m17.851s
>
> local through file url
> real 12m33.247s
> user 0m36.833s
> sys 0m19.608s
>
> The difference in time to my original client-server test are caused by
> the slower harddisks on the client system.
Very interesting tests. But this part of code was strongly improved
after 1.3.x release (in client side). So will be good if you repeat
your tests with trunk code. I can send you compiled subversion
binaries for trunk if you cannot build yourself.
--
Ivan Zhakov
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@silver-style.com>.
>> Subversion 1.1.4
>
> Oh. This is worth looking into if you can reproduce the results with a
> 1.3 server. Otherwise it's just a waste of time; we do know that some
> operations are slower against an older server.
I doubt that this has anything to do with the server. For 75% of the
time there isn't even a connection to the server. For completeness the
test with a version 1.3.0 server.
Commit of an 1GB file into an empty repository.
Client: svn 1.3.0, Windows XP, NTFS
Server: svn 1.3.0, Windows XP, NTFS, svnserve
$ time svn commit -m "test"
Hinzuf. (bin) test.tmp
šbertrage Daten .
Revision 1 bertragen.
real 34m42.985s
user 0m0.031s
sys 0m0.000s
Unfortunately I can not place an updated version of subversion on the
debian box.
> You don't say if your file has svn:eol-style or svn:keywords
properties. I'm assuming it does.
Uh, no? I have an empty repository, co that, put an 1GB file into the wc
and commit that file. No properties that I know off. svn proplist shows
only svn:mime-type. svn:eol-style or svn:keywords wouldn't make much
sense as the file contains data fetched from /dev/urandom.
Sebastian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@gmx.net>.
> > real 34m42.985s
> > user 0m0.031s
> > sys 0m0.000s
>
> I don't know much about Windows (that looks like a Unix time?) but
> your timings appear to show almost no CPU usage in the client.
It is cygwin actually.
> I guess the client is spending all it's time waiting for IO, which means
> the problem is either the server or the network.
You are right on the io, but not network or server but local disc io. As I
said in an earlier post, most of the used time the server doesn't see a
connection or any traffic.
> I suspect the server
> is your bottleneck, it takes quite a bit of CPU to decode the svndiff
> received from the client and then encode it for the repository. (I'm
> a bit surprised that the client can encode a 1GB file using only
> 0.031s of CPU.)
I do not know if there is much to do for a new file. The diff would be quite
simple: everything changed. I repeat myself but the server sees for more
than 75% of the time no traffic or connection at all. Additional to that the
server can handle the send data way faster than it is transmitted. The
client sends the data at a rate of 6MB/s. That would be under 3min for 1GB
worth of data.
I have no idea how the time command gets the cpu time, but as the client
does nothing more than wait for the harddisk the results are not very
surprising. Because the cpu time measuring on windows is a bit odd the
values may be skewed down. But not in orders of magnitude.
To bring this forward besides the block sizes on file operations: I would
like to have a list of actions done by the client that match the observed
file operations.
If you have ideas for any further experiments I can conduct I am glad to
help.
Sebastian
--
Echte DSL-Flatrate dauerhaft f�r 0,- Euro*!
"Feel free" mit GMX DSL! http://www.gmx.net/de/go/dsl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Michael Sinz <Mi...@sinz.org>.
On 3/9/06, Philip Martin <ph...@codematters.co.uk> wrote:
> Sebastian Tusk <se...@gmx.net> writes:
>
> > I doubt that this has anything to do with the server.
>
> Hmmm...
>
> > For 75% of the
> > time there isn't even a connection to the server. For completeness the
> > test with a version 1.3.0 server.
> >
> > Commit of an 1GB file into an empty repository.
> >
> > Client: svn 1.3.0, Windows XP, NTFS
> > Server: svn 1.3.0, Windows XP, NTFS, svnserve
> >
> > $ time svn commit -m "test"
> > Hinzuf. (bin) test.tmp
> > šbertrage Daten .
> > Revision 1 bertragen.
> >
> > real 34m42.985s
> > user 0m0.031s
> > sys 0m0.000s
>
> I don't know much about Windows (that looks like a Unix time?) but
> your timings appear to show almost no CPU usage in the client. I
> guess the client is spending all it's time waiting for IO, which means
> the problem is either the server or the network. I suspect the server
> is your bottleneck, it takes quite a bit of CPU to decode the svndiff
> received from the client and then encode it for the repository. (I'm
> a bit surprised that the client can encode a 1GB file using only
> 0.031s of CPU.)
Actually, the accounting in Windows is not too swift. This looks like
local I/O problems, not network I/O problems. You can tell a bit more
if you run a windows specific tool to monitor performance.
(Both the network side and the local side)
BTW - Don't ask me about windows - I try to stay as far away from
it as possible. Unfortunately it is never far enough :-(
--
Michael Sinz Technology and Engineering Director/Consultant
"Starting Startups" mailto:Michael.Sinz@sinz.org
My place on the web http://www.sinz.org/Michael.Sinz
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Philip Martin <ph...@codematters.co.uk>.
Sebastian Tusk <se...@gmx.net> writes:
> I doubt that this has anything to do with the server.
Hmmm...
> For 75% of the
> time there isn't even a connection to the server. For completeness the
> test with a version 1.3.0 server.
>
> Commit of an 1GB file into an empty repository.
>
> Client: svn 1.3.0, Windows XP, NTFS
> Server: svn 1.3.0, Windows XP, NTFS, svnserve
>
> $ time svn commit -m "test"
> Hinzuf. (bin) test.tmp
> šbertrage Daten .
> Revision 1 bertragen.
>
> real 34m42.985s
> user 0m0.031s
> sys 0m0.000s
I don't know much about Windows (that looks like a Unix time?) but
your timings appear to show almost no CPU usage in the client. I
guess the client is spending all it's time waiting for IO, which means
the problem is either the server or the network. I suspect the server
is your bottleneck, it takes quite a bit of CPU to decode the svndiff
received from the client and then encode it for the repository. (I'm
a bit surprised that the client can encode a 1GB file using only
0.031s of CPU.)
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@gmx.net>.
>> Subversion 1.1.4
>
> Oh. This is worth looking into if you can reproduce the results with a
> 1.3 server. Otherwise it's just a waste of time; we do know that some
> operations are slower against an older server.
I doubt that this has anything to do with the server. For 75% of the
time there isn't even a connection to the server. For completeness the
test with a version 1.3.0 server.
Commit of an 1GB file into an empty repository.
Client: svn 1.3.0, Windows XP, NTFS
Server: svn 1.3.0, Windows XP, NTFS, svnserve
$ time svn commit -m "test"
Hinzuf. (bin) test.tmp
šbertrage Daten .
Revision 1 bertragen.
real 34m42.985s
user 0m0.031s
sys 0m0.000s
Unfortunately I can not place an updated version of subversion on the
debian box.
> You don't say if your file has svn:eol-style or svn:keywords
properties. I'm assuming it does.
Uh, no? I have an empty repository, co that, put an 1GB file into the wc
and commit that file. No properties that I know off. svn proplist shows
only svn:mime-type. svn:eol-style or svn:keywords wouldn't make much
sense as the file contains data fetched from /dev/urandom.
Sebastian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Branko Čibej <br...@xbc.nu>.
Sebastian Tusk wrote:
>> Subversion client:
>> * Subversion v1.3.0
>> Subversion server:
> Subversion 1.1.4
Oh. This is worth looking into if you can reproduce the results with a
1.3 server. Otherwise it's just a waste of time; we do know that some
operations are slower against an older server.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@gmx.net>.
Julian Foad wrote:
> Sebastian Tusk wrote:
>
>> I did some tests with a repository that contains a single file sized
>> 686 MB. I used the svn client version 1.3.0 on a windows machine.
>>
>> svn commit 1587s
>> svn export 136s
>> svn co 395s
>>
>> To me this figures appeared way to high.
>
>
> You're right: these figures do appear to be too high, at least for
> commit and checkout. I don't suppose it is that slow on all set-ups, so
> it might help one of us to investigate the problem if you could you give
> some more details about the system:
>
> Subversion client:
> * Subversion v1.3.0
> * running on what version of MS Windows?
> * what filesystem type (FAT, NTFS, etc.)?
Windows XP Prof
NTFS
> Subversion server:
> * which version?
> * which type of repository (FSFS or BDB)?
> * which type of Subversion server (local, svnserve, Apache/DAV)?
> * running on what OS?
> * what filesystem type (FAT, NTFS, etc.)?
Subversion 1.1.4
BDB
Apache/DAV
Debian 3.1
ext3
Server and client are connected with 100Mbit Ethernet.
As comparison here the results if I am doing the commit of a 1GB file on
the server
local through dav
real 12m5.226s
user 2m29.560s
sys 0m17.851s
local through file url
real 12m33.247s
user 0m36.833s
sys 0m19.608s
The difference in time to my original client-server test are caused by
the slower harddisks on the client system.
Sebastian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Julian Foad <ju...@btopenworld.com>.
Sebastian Tusk wrote:
> I did some tests with a repository that contains a single file sized 686
> MB. I used the svn client version 1.3.0 on a windows machine.
>
> svn commit 1587s
> svn export 136s
> svn co 395s
>
> To me this figures appeared way to high.
You're right: these figures do appear to be too high, at least for commit and
checkout. I don't suppose it is that slow on all set-ups, so it might help one
of us to investigate the problem if you could you give some more details about
the system:
Subversion client:
* Subversion v1.3.0
* running on what version of MS Windows?
* what filesystem type (FAT, NTFS, etc.)?
Subversion server:
* which version?
* which type of repository (FSFS or BDB)?
* which type of Subversion server (local, svnserve, Apache/DAV)?
* running on what OS?
* what filesystem type (FAT, NTFS, etc.)?
Thanks for letting us know of the problem. Maybe someone will be able to
determine which particular circumstances give the worst performance, and make
it better.
- Julian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Branko Čibej <br...@xbc.nu>.
Justin Erenkrantz wrote:
> On 3/13/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
>
>> It needs to read the whole target, but not the whole source. The
>> delta algorithm will read one Window (100 kB) at a time from source
>> and target. If the target is longer than the source, the source is not
>> read to its end. (Note that the delta doesn't contain delete instructions.)
>>
>
> Why isn't APR_BUFFERED turned on?
It's not 100k-at-a-time to achieve buffering; that's simply the delta
window size. Our current delta implementation(s) require fixed-size
delta windows.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 3/13/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> It needs to read the whole target, but not the whole source. The
> delta algorithm will read one Window (100 kB) at a time from source
> and target. If the target is longer than the source, the source is not
> read to its end. (Note that the delta doesn't contain delete instructions.)
Why isn't APR_BUFFERED turned on? -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Ivan Zhakov writes:
> On 3/13/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > Ivan Zhakov writes:
> > > On 3/13/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > > > Maybe it is a
> > > > good idea to get rid of the read_all parameter of the
> > > > svn_stream_content_same function and instead create a new stream that
> > > > will "drain" its underlying stream when it is being closed. Thoughts?
> > > Yeah, I had similiar idea recently. But in this case we need extra
> > > allocations of read buffer. I don't know how it decrease performance.
> >
> > Keep a flag whether the end of stream was reached. On close, if EOS
> > wasn't reached, just allocate a buffer and drain the stream. So you
> > only need the buffer if the reading was stopped early.
> >
> Or might be add this flag to svn_stream_checksummed() ?
I added this capability to svn_stream_checksummed mostly because I don't see who else would need to.
> I will be busy in near time therefore I cannot fix it.
>
r18944. Please have a look if you have time. The last thing here is
to transfer the md5 of the new text base from transmit_text_deltas to
the post commit processing. Working on that now.
Thank you for working in this area,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Ivan Zhakov <ch...@gmail.com>.
On 3/13/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Ivan Zhakov writes:
> > On 3/13/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > > Maybe it is a
> > > good idea to get rid of the read_all parameter of the
> > > svn_stream_content_same function and instead create a new stream that
> > > will "drain" its underlying stream when it is being closed. Thoughts?
> > Yeah, I had similiar idea recently. But in this case we need extra
> > allocations of read buffer. I don't know how it decrease performance.
>
> Keep a flag whether the end of stream was reached. On close, if EOS
> wasn't reached, just allocate a buffer and drain the stream. So you
> only need the buffer if the reading was stopped early.
>
Or might be add this flag to svn_stream_checksummed() ?
I will be busy in near time therefore I cannot fix it.
--
Ivan Zhakov
Re: export, checkout, commit performance
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Ivan Zhakov writes:
> On 3/13/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
>
> > Also, we need to make sure that the whole text base is
> > read. I'm not sure if the txdelta generator will do that if the
> > working file is considerably shorter than the text base.
> I can't figure how txdelta can send delta without reading whole stream?
>
It needs to read the whole target, but not the whole source. The
delta algorithm will read one Window (100 kB) at a time from source
and target. If the target is longer than the source, the source is not
read to its end. (Note that the delta doesn't contain delete instructions.)
> > Maybe it is a
> > good idea to get rid of the read_all parameter of the
> > svn_stream_content_same function and instead create a new stream that
> > will "drain" its underlying stream when it is being closed. Thoughts?
> Yeah, I had similiar idea recently. But in this case we need extra
> allocations of read buffer. I don't know how it decrease performance.
Keep a flag whether the end of stream was reached. On close, if EOS
wasn't reached, just allocate a buffer and drain the stream. So you
only need the buffer if the reading was stopped early.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Ivan Zhakov <ch...@gmail.com>.
On 3/13/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Ivan Zhakov writes:
> > >
> > > Also I mention there is additional checksum checking in case if not
> > > fulltext deltas sent, which also can be avoided if move it before
> > > editor->close_file() call. This mean that checksum will be checked
> > > after transmission, but before closing file. I consider this is
> > > acceptable.
> > >
> > After detailed analysis I found that is impossible -- text_editor
> > wants base checksum BEFORE receiving delta. But we can calc checksum
> > during copy file to temp base.
>
> The textbase is not copied to a temporary location (only the working
> file that *will become* the text base). But, you can use the checksum
> from the entries file and verify that on the fly when the text delta
> is being transmitted. We need to be careful with compatibility with
> WCs created before 2003-02-13, though, but you should be able to use
> the length of the checksum string to detect if it is base64
> encoded.
Hmm, interesting idea. I'll investigate how complex it will be. Also I
can use WC_FORMAT for detecting old working copies.
> Also, we need to make sure that the whole text base is
> read. I'm not sure if the txdelta generator will do that if the
> working file is considerably shorter than the text base.
I can't figure how txdelta can send delta without reading whole stream?
> Maybe it is a
> good idea to get rid of the read_all parameter of the
> svn_stream_content_same function and instead create a new stream that
> will "drain" its underlying stream when it is being closed. Thoughts?
Yeah, I had similiar idea recently. But in this case we need extra
allocations of read buffer. I don't know how it decrease performance.
>
> > Step (5) odd for me. We check for modifications to make decision about
> > right timestamp. But IMHO presence of temporary file already means
> > text modifications, is it correct?
>
> This checks if the working file was modified *after* the temporary
> file was created. I.e., you start a commit of a large file, which gets
> copied into a temporary file. Something modifies the working file and
> after that, the commit is finalized. There obviously is a race here
> (the file might be modified after you start checking for identity and
> before the timestamp is set). I *think* the intent is to avoid
> undetected modifications as far as possible.
Thanks for explanations.
>
> An alternative might be to fetch the timestamp of the working file
> when copying it in the svn_wc_transmit_text_deltas function and use
> that timestamp in the entries file in the post-commit processing. I'm
> not sure how that would interact with the read-only and executable
> permission changes post-commit, though.
>
--
Ivan Zhakov
Re: base64 encoded md5 digests in entries file
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Julian Foad writes:
> Peter N. Lundblad wrote:
> > Well, if people agree we don't need this compat hack, I'll not bother.
>
> I have been using my main Subversion source working copy continuously since
> those olden days, and today "svn commit" choked on one of the old checksums
> when committing a file that hasn't changed since then
> (notes/fs-improvements.txt). I have been encountering another such file once
> in every few months since whenever the support for the old style was dropped.
Oh, so this means that we've already dropped part of this support
earlier. Well, then my recent commit didn't actually make things worse.
> "commit" and "revert" are the only two operations that I have noticed failing
> on these old files. I'm not sure whether any other operations explicitly
> support old-style checksums, or if other operations just don't look at them.
>
> My work-around is to temporarily save and remove the local changes, then update
Another workaround should be to just remove the ckecksum attribute
from the entries file, because that's optional.
I'll not do anything about my commit in svn_wc_transmit_text_deltas2
until we decide to reintroduce the compat hack, but since this seems
like more work, I don't think it is worht it. I agree with Zhakov, it
starts to be time to drop pre-1.0 compatibility hacks.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: base64 encoded md5 digests in entries file
Posted by Julian Foad <ju...@btopenworld.com>.
Peter N. Lundblad wrote:
> Ivan Zhakov writes:
> > I really don't understand why we should support these old working
> > copies because they created BEFORE 1.0 was released. So we shouldn't
> > support it, if understand our compability policy correctly.
I agree that we don't NEED to support them, but see below. (Our compatibility
policy may say that we do not need to support them, but it never says that we
should not (ought not to) support the old ways.)
> Well, if people agree we don't need this compat hack, I'll not bother.
I have been using my main Subversion source working copy continuously since
those olden days, and today "svn commit" choked on one of the old checksums
when committing a file that hasn't changed since then
(notes/fs-improvements.txt). I have been encountering another such file once
in every few months since whenever the support for the old style was dropped.
"commit" and "revert" are the only two operations that I have noticed failing
on these old files. I'm not sure whether any other operations explicitly
support old-style checksums, or if other operations just don't look at them.
My work-around is to temporarily save and remove the local changes, then update
the file to r0 (thus deleting it) and then to HEAD (thus getting a new-style
checksum), and re-applying the local changes. I think it's perfectly
acceptable to have to do something like that once in a while for someone who
has kept a WC around since way before v1.0.
> OTOH, there was no upgrade path in this case and adding two lines to be
> friendly doesn't seem too much:-) I really don't have a strong opinion here.
If the whole of Subversion (commit, revert and all other operations) can
support those old checksums with just a few lines of compatibility code, I'd
say make it compatible, because we know we do have some users who started
before v1.0 and providing a smooth upgrade for users is a very important factor
in their overall impression of the software. However, there are rather few of
them and they know that pre-1.0 incompatibility is acceptable, so if it
involves any more than a few lines of code (to support it throughout the code
base), strip it out.
- Julian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: base64 encoded md5 digests in entries file (was: Re: export, checkout, commit performance)
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Ivan Zhakov writes:
> I really don't understand why we should support these old working
> copies because they created BEFORE 1.0 was released. So we shouldn't
> support it, if understand our compability policy correctly.
>
Well, if people agree we don't need this compat hack, I'll not bother.
OTOH, there was no upgrade path in this case and adding two lines to be
friendly doesn't seem too much:-) I really don't have a strong opinion here.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: base64 encoded md5 digests in entries file (was: Re: export, checkout, commit performance)
Posted by Ivan Zhakov <ch...@gmail.com>.
On 3/19/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Peter N. Lundblad writes:
> > Malcolm Rowe writes:
> > > On Fri, Mar 17, 2006 at 11:16:19PM +0100, Peter N. Lundblad wrote:
> > > > Malcolm Rowe writes:
> > > Even better! Looking around for any other evidence of the old behaviour,
> > > I see that (struct svn_wc_entry_t).checksum is currently documented as:
> > >
> > > /** base64-encoded checksum for the untranslated text base file,
> > > * can be @c NULL for backwards compatibility.
> > > */
> > > const char *checksum;
> > >
> > > (in include/svn_wc.h).
> > >
> > > I presume that's wrong, but I don't know what the best way to describe
> > > the checksum is - 'ASCII-encoded MD5 checksum'? Not very clear, imo.
> > >
> >
> > I think "hex md5 checksum" would be clear.
> >
>
> Bah! If we go back to the original problem, namely whether to
> preserve backwards compatibility for entries files created before
> early 2003. Earlier, I thought these were going to be auto-upgraded
> to use hex checksums instead of base64 ditto. Unfortunately, this is
> not the case because even if the entries file is rewritten, the
> contents of the checksum attribute is preserved asis.
>
> I guess I'll just check the length of the checksum entry field in svn
> _wc_transmit_textdeltas2 and punt if it is not the expected length of
> a hex checksum.
>
> Thoughts, anyone?
>
I really don't understand why we should support these old working
copies because they created BEFORE 1.0 was released. So we shouldn't
support it, if understand our compability policy correctly.
--
Ivan Zhakov
base64 encoded md5 digests in entries file (was: Re: export, checkout, commit performance)
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Peter N. Lundblad writes:
> Malcolm Rowe writes:
> > On Fri, Mar 17, 2006 at 11:16:19PM +0100, Peter N. Lundblad wrote:
> > > Malcolm Rowe writes:
> > Even better! Looking around for any other evidence of the old behaviour,
> > I see that (struct svn_wc_entry_t).checksum is currently documented as:
> >
> > /** base64-encoded checksum for the untranslated text base file,
> > * can be @c NULL for backwards compatibility.
> > */
> > const char *checksum;
> >
> > (in include/svn_wc.h).
> >
> > I presume that's wrong, but I don't know what the best way to describe
> > the checksum is - 'ASCII-encoded MD5 checksum'? Not very clear, imo.
> >
>
> I think "hex md5 checksum" would be clear.
>
Bah! If we go back to the original problem, namely whether to
preserve backwards compatibility for entries files created before
early 2003. Earlier, I thought these were going to be auto-upgraded
to use hex checksums instead of base64 ditto. Unfortunately, this is
not the case because even if the entries file is rewritten, the
contents of the checksum attribute is preserved asis.
I guess I'll just check the length of the checksum entry field in svn
_wc_transmit_textdeltas2 and punt if it is not the expected length of
a hex checksum.
Thoughts, anyone?
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Malcolm Rowe writes:
> On Fri, Mar 17, 2006 at 11:16:19PM +0100, Peter N. Lundblad wrote:
> > Malcolm Rowe writes:
> Even better! Looking around for any other evidence of the old behaviour,
> I see that (struct svn_wc_entry_t).checksum is currently documented as:
>
> /** base64-encoded checksum for the untranslated text base file,
> * can be @c NULL for backwards compatibility.
> */
> const char *checksum;
>
> (in include/svn_wc.h).
>
> I presume that's wrong, but I don't know what the best way to describe
> the checksum is - 'ASCII-encoded MD5 checksum'? Not very clear, imo.
>
I think "hex md5 checksum" would be clear.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Fri, Mar 17, 2006 at 11:16:19PM +0100, Peter N. Lundblad wrote:
> Malcolm Rowe writes:
> > This is r4374, yes? I know that wc's are long-lived, but do we _really_
> > need to be able to read wc's that haven't been updated in the last
> > three years, since 0.17.x? (Or am I wrong in thinking we'll rewrite
> > the checksum whenever we rewrite the entries file?)
> >
>
> Well, since the function we're talking about requires a write lock,
> and that will have upgraded the WC to format version 6, which in turn
> will have rewritten the entries file, I conclude that you're right.
> Dropping that compat hack now.
>
Even better! Looking around for any other evidence of the old behaviour,
I see that (struct svn_wc_entry_t).checksum is currently documented as:
/** base64-encoded checksum for the untranslated text base file,
* can be @c NULL for backwards compatibility.
*/
const char *checksum;
(in include/svn_wc.h).
I presume that's wrong, but I don't know what the best way to describe
the checksum is - 'ASCII-encoded MD5 checksum'? Not very clear, imo.
Regards,
Malcolm
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Malcolm Rowe writes:
> On Mon, Mar 13, 2006 at 03:28:03PM +0100, Peter N. Lundblad wrote:
> > We need to be careful with compatibility with
> > WCs created before 2003-02-13, though, but you should be able to use
> > the length of the checksum string to detect if it is base64
> > encoded.
>
> This is r4374, yes? I know that wc's are long-lived, but do we _really_
> need to be able to read wc's that haven't been updated in the last
> three years, since 0.17.x? (Or am I wrong in thinking we'll rewrite
> the checksum whenever we rewrite the entries file?)
>
Well, since the function we're talking about requires a write lock,
and that will have upgraded the WC to format version 6, which in turn
will have rewritten the entries file, I conclude that you're right.
Dropping that compat hack now.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Mon, Mar 13, 2006 at 03:28:03PM +0100, Peter N. Lundblad wrote:
> We need to be careful with compatibility with
> WCs created before 2003-02-13, though, but you should be able to use
> the length of the checksum string to detect if it is base64
> encoded.
This is r4374, yes? I know that wc's are long-lived, but do we _really_
need to be able to read wc's that haven't been updated in the last
three years, since 0.17.x? (Or am I wrong in thinking we'll rewrite
the checksum whenever we rewrite the entries file?)
Regards,
Malcolm
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Philip Martin writes:
> "Peter N. Lundblad" <pe...@famlundblad.se> writes:
>
> > An alternative might be to fetch the timestamp of the working file
> > when copying it in the svn_wc_transmit_text_deltas function and use
> > that timestamp in the entries file in the post-commit processing.
>
> Nearly right. We could fetch/store the working timestamp when copying
> and then compare the stored and working values during post-commit and
> only do the byte-for-byte comparison if the timestamp has changed.
> This is essentially the same algorithm used to detect other text
> changes.
>
OK, that avoids commands such as status doing the expensive check each
time afterwards. I'll look into this and also keeping the new base
checksum so we don't calculate it twice as we do now.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Philip Martin <ph...@codematters.co.uk>.
"Peter N. Lundblad" <pe...@famlundblad.se> writes:
> An alternative might be to fetch the timestamp of the working file
> when copying it in the svn_wc_transmit_text_deltas function and use
> that timestamp in the entries file in the post-commit processing.
Nearly right. We could fetch/store the working timestamp when copying
and then compare the stored and working values during post-commit and
only do the byte-for-byte comparison if the timestamp has changed.
This is essentially the same algorithm used to detect other text
changes.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Ivan Zhakov writes:
> >
> > Also I mention there is additional checksum checking in case if not
> > fulltext deltas sent, which also can be avoided if move it before
> > editor->close_file() call. This mean that checksum will be checked
> > after transmission, but before closing file. I consider this is
> > acceptable.
> >
> After detailed analysis I found that is impossible -- text_editor
> wants base checksum BEFORE receiving delta. But we can calc checksum
> during copy file to temp base.
The textbase is not copied to a temporary location (only the working
file that *will become* the text base). But, you can use the checksum
from the entries file and verify that on the fly when the text delta
is being transmitted. We need to be careful with compatibility with
WCs created before 2003-02-13, though, but you should be able to use
the length of the checksum string to detect if it is base64
encoded. Also, we need to make sure that the whole text base is
read. I'm not sure if the txdelta generator will do that if the
working file is considerably shorter than the text base. Maybe it is a
good idea to get rid of the read_all parameter of the
svn_stream_content_same function and instead create a new stream that
will "drain" its underlying stream when it is being closed. Thoughts?
> Step (5) odd for me. We check for modifications to make decision about
> right timestamp. But IMHO presence of temporary file already means
> text modifications, is it correct?
This checks if the working file was modified *after* the temporary
file was created. I.e., you start a commit of a large file, which gets
copied into a temporary file. Something modifies the working file and
after that, the commit is finalized. There obviously is a race here
(the file might be modified after you start checking for identity and
before the timestamp is set). I *think* the intent is to avoid
undetected modifications as far as possible.
An alternative might be to fetch the timestamp of the working file
when copying it in the svn_wc_transmit_text_deltas function and use
that timestamp in the entries file in the post-commit processing. I'm
not sure how that would interact with the read-only and executable
permission changes post-commit, though.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Ivan Zhakov <ch...@gmail.com>.
On 3/10/06, Ivan Zhakov <ch...@gmail.com> wrote:
> On 3/10/06, Sebastian Tusk <se...@gmx.net> wrote:
> > I repeated the test from my original post.
> >
> > Add/commit of a single file named "test".
> > server: stock 1.3.0 svnserve, Windows XP, NTFS
> > client: trunk build, Windows XP, NTFS
> >
> > 1. copy, read test, write .svn\tmp\text-base\test.svn-base.tmp (512b blocks)
> > stack: apr_file_copy <- svn_io_copy_file <- svn_wc_transmit_text_deltas
> > <- svn_client__do_commit
> >
> > 2. transmit, read .svn\tmp\text-base\test.svn-base (100kb blocks)
> > stack: svn_txdelta__vdelta <- compute_window <- svn_txdelta_next_window
> > <- svn_txdelta_send_txstream <- svn_wc_transmit_text_deltas <-
> > svn_client__do_commit
> >
> > 3. hashing, read .svn\tmp\text-base\test.svn-base (512b blocks)
> > stack: svn_io_file_checksum <- svn_wc_transmit_text_deltas <-
> > svn_client__do_commit
> >
> > 4. hashing, read .svn\tmp\text-base\test.svn-base (512b blocks)
> > stack: svn_io_file_checksum <- svn_wc_process_committed2
> >
> > 5. compare, read test and .svn\tmp\text-base\test.svn-base simultaneous
> > (512b blocks)
> > stack: contents_identical_p <- svn_io_files_contents_same_p <-
> > compare_and_verify <- svn_wc__versioned_file_modcheck <-
> > log_do_committed <- start_handler <- expat_start_handler <- doContent <-
> > contentProcessor <- XML_ParseBuffer <- XML_Parse <- svn_xml_parse <-
> > run_log <- svn_wc__run_log <- svn_wc_process_committed2
> >
> >
> > Most obvious problem is that test.svn-base gets hashed two times. Is the
> > comparison (point 5) necessary?
> Steps (2) and (3) can be ease merged with new stream translation API,
> with change like this (don't look to code style, it is only for
> testing!):
> Index: subversion/libsvn_wc/adm_crawler.c
> ===================================================================
> --- subversion/libsvn_wc/adm_crawler.c (revision 18809)
> +++ subversion/libsvn_wc/adm_crawler.c (working copy)
> @@ -714,8 +714,9 @@
> apr_file_t *localfile = NULL;
> apr_file_t *basefile = NULL;
> const char *base_digest_hex = NULL;
> - unsigned char digest[APR_MD5_DIGESTSIZE];
> -
> + unsigned char *digest;
> + svn_stream_t *local_stream;
> +
> /* Make an untranslated copy of the working file in the
> administrative tmp area because a) we want this to work even if
> someone changes the working file while we're generating the
> @@ -823,9 +824,11 @@
>
> /* Create a text-delta stream object that pulls data out of the two
> files. */
> + local_stream =
> svn_stream_checksummed(svn_stream_from_aprfile(localfile, pool),
> + &digest, NULL, pool);
> svn_txdelta(&txdelta_stream,
> svn_stream_from_aprfile(basefile, pool),
> - svn_stream_from_aprfile(localfile, pool),
> + local_stream,
> pool);
>
> /* Pull windows from the delta stream and feed to the consumer. */
> @@ -834,18 +837,11 @@
>
> /* Close the two files */
> SVN_ERR(svn_io_file_close(localfile, pool));
> + SVN_ERR(svn_stream_close(local_stream));
>
> if (basefile)
> SVN_ERR(svn_wc__close_text_base(basefile, path, 0, pool));
>
> - /* ### This is a pity. tmp_base was created with svn_io_copy_file()
> - above, which uses apr_file_copy(), which probably called
> - apr_file_transfer_contents(), which ran over every byte of the
> - file and therefore could have computed a checksum effortlessly.
> - But we're not about to change the interface of apr_file_copy(),
> - so we'll have to run over the bytes again... */
> - SVN_ERR(svn_io_file_checksum(digest, tmp_base, pool));
> -
> /* Close the file baton, and get outta here. */
> return editor->close_file
> (file_baton, svn_md5_digest_to_cstring(digest, pool), pool);
>
Commited in r18867. Thanks, Sebastian. Will be great if you made such
analys for other operations like checkout, update and etc.
>
> Also I mention there is additional checksum checking in case if not
> fulltext deltas sent, which also can be avoided if move it before
> editor->close_file() call. This mean that checksum will be checked
> after transmission, but before closing file. I consider this is
> acceptable.
>
After detailed analysis I found that is impossible -- text_editor
wants base checksum BEFORE receiving delta. But we can calc checksum
during copy file to temp base.
Step (5) odd for me. We check for modifications to make decision about
right timestamp. But IMHO presence of temporary file already means
text modifications, is it correct?
--
Ivan Zhakov
Re: export, checkout, commit performance
Posted by Ivan Zhakov <ch...@gmail.com>.
On 3/11/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Ivan Zhakov writes:
> > On 3/10/06, Sebastian Tusk <se...@gmx.net> wrote:
> > >
> > > Most obvious problem is that test.svn-base gets hashed two times. Is the
> > > comparison (point 5) necessary?
> > Steps (2) and (3) can be ease merged with new stream translation API,
> > with change like this (don't look to code style, it is only for
> > testing!):
> ...
>
> Looks good, and even if it doesn't increase performance very much, I think
> it is good because it increases parallellization between client and server.
It should improve performance dramaticly when file larger system disk cache.
>
> > Also I mention there is additional checksum checking in case if not
> > fulltext deltas sent, which also can be avoided if move it before
> > editor->close_file() call. This mean that checksum will be checked
> > after transmission, but before closing file. I consider this is
> > acceptable.
> >
>
> Agreed. The important thing is that the transaction won't be
> committed in the end. The server might complain about corrupt
> svndiff, but only if the text-base is very corrupt (i.e. longer than
> it should be for exmple).
>
> Will you work on this?
Yes, I am going to commit these two changes in Monday. If you have
time, you can commit it of course. Might be better use owned streams
to close file automaticly.
--
Ivan Zhakov
Re: export, checkout, commit performance
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Ivan Zhakov writes:
> On 3/10/06, Sebastian Tusk <se...@gmx.net> wrote:
> >
> > Most obvious problem is that test.svn-base gets hashed two times. Is the
> > comparison (point 5) necessary?
> Steps (2) and (3) can be ease merged with new stream translation API,
> with change like this (don't look to code style, it is only for
> testing!):
...
Looks good, and even if it doesn't increase performance very much, I think
it is good because it increases parallellization between client and server.
> Also I mention there is additional checksum checking in case if not
> fulltext deltas sent, which also can be avoided if move it before
> editor->close_file() call. This mean that checksum will be checked
> after transmission, but before closing file. I consider this is
> acceptable.
>
Agreed. The important thing is that the transaction won't be
committed in the end. The server might complain about corrupt
svndiff, but only if the text-base is very corrupt (i.e. longer than
it should be for exmple).
Will you work on this?
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Ivan Zhakov <ch...@gmail.com>.
On 3/10/06, Sebastian Tusk <se...@gmx.net> wrote:
> I repeated the test from my original post.
>
> Add/commit of a single file named "test".
> server: stock 1.3.0 svnserve, Windows XP, NTFS
> client: trunk build, Windows XP, NTFS
>
> 1. copy, read test, write .svn\tmp\text-base\test.svn-base.tmp (512b blocks)
> stack: apr_file_copy <- svn_io_copy_file <- svn_wc_transmit_text_deltas
> <- svn_client__do_commit
>
> 2. transmit, read .svn\tmp\text-base\test.svn-base (100kb blocks)
> stack: svn_txdelta__vdelta <- compute_window <- svn_txdelta_next_window
> <- svn_txdelta_send_txstream <- svn_wc_transmit_text_deltas <-
> svn_client__do_commit
>
> 3. hashing, read .svn\tmp\text-base\test.svn-base (512b blocks)
> stack: svn_io_file_checksum <- svn_wc_transmit_text_deltas <-
> svn_client__do_commit
>
> 4. hashing, read .svn\tmp\text-base\test.svn-base (512b blocks)
> stack: svn_io_file_checksum <- svn_wc_process_committed2
>
> 5. compare, read test and .svn\tmp\text-base\test.svn-base simultaneous
> (512b blocks)
> stack: contents_identical_p <- svn_io_files_contents_same_p <-
> compare_and_verify <- svn_wc__versioned_file_modcheck <-
> log_do_committed <- start_handler <- expat_start_handler <- doContent <-
> contentProcessor <- XML_ParseBuffer <- XML_Parse <- svn_xml_parse <-
> run_log <- svn_wc__run_log <- svn_wc_process_committed2
>
>
> Most obvious problem is that test.svn-base gets hashed two times. Is the
> comparison (point 5) necessary?
Steps (2) and (3) can be ease merged with new stream translation API,
with change like this (don't look to code style, it is only for
testing!):
Index: subversion/libsvn_wc/adm_crawler.c
===================================================================
--- subversion/libsvn_wc/adm_crawler.c (revision 18809)
+++ subversion/libsvn_wc/adm_crawler.c (working copy)
@@ -714,8 +714,9 @@
apr_file_t *localfile = NULL;
apr_file_t *basefile = NULL;
const char *base_digest_hex = NULL;
- unsigned char digest[APR_MD5_DIGESTSIZE];
-
+ unsigned char *digest;
+ svn_stream_t *local_stream;
+
/* Make an untranslated copy of the working file in the
administrative tmp area because a) we want this to work even if
someone changes the working file while we're generating the
@@ -823,9 +824,11 @@
/* Create a text-delta stream object that pulls data out of the two
files. */
+ local_stream =
svn_stream_checksummed(svn_stream_from_aprfile(localfile, pool),
+ &digest, NULL, pool);
svn_txdelta(&txdelta_stream,
svn_stream_from_aprfile(basefile, pool),
- svn_stream_from_aprfile(localfile, pool),
+ local_stream,
pool);
/* Pull windows from the delta stream and feed to the consumer. */
@@ -834,18 +837,11 @@
/* Close the two files */
SVN_ERR(svn_io_file_close(localfile, pool));
+ SVN_ERR(svn_stream_close(local_stream));
if (basefile)
SVN_ERR(svn_wc__close_text_base(basefile, path, 0, pool));
- /* ### This is a pity. tmp_base was created with svn_io_copy_file()
- above, which uses apr_file_copy(), which probably called
- apr_file_transfer_contents(), which ran over every byte of the
- file and therefore could have computed a checksum effortlessly.
- But we're not about to change the interface of apr_file_copy(),
- so we'll have to run over the bytes again... */
- SVN_ERR(svn_io_file_checksum(digest, tmp_base, pool));
-
/* Close the file baton, and get outta here. */
return editor->close_file
(file_baton, svn_md5_digest_to_cstring(digest, pool), pool);
Also I mention there is additional checksum checking in case if not
fulltext deltas sent, which also can be avoided if move it before
editor->close_file() call. This mean that checksum will be checked
after transmission, but before closing file. I consider this is
acceptable.
--
Ivan Zhakov
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@gmx.net>.
> Ah, seems like zhakov (I assume it was his binaries you used) just
> missed Branko's commit that eliminated the 512-byte uses (except the
> above). Here are binaries built after this commit. It *should* use 16K
> blocks when checksumming and comparing files.
It seems I missed these patches with my build. Here the test results for
a 100 MB file added and committed.
Build without patches: 1min 25s
Build with patches: 1min 22s
Not much of a difference.
These times do not grow linear with the file size. A 1GB file takes
29min 27s with the latest patches. I guess the 100MB file gets cached by
the os.
In any case it is a lot better than 1.3.0 where a 680 MB files takes
almost the same time.
Sebastian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@silver-style.com>.
> > 1. copy, read test, write .svn\tmp\text-base\test.svn-base.tmp (512b blocks)
> > stack: apr_file_copy <- svn_io_copy_file <- svn_wc_transmit_text_deltas
> > <- svn_client__do_commit
>
> Ah, seems like zhakov (I assume it was his binaries you used) just
> missed Branko's commit that eliminated the 512-byte uses (except the
> above). Here are binaries built after this commit. It *should* use 16K
> blocks when checksumming and comparing files.
No, it is a build by myself directly from trunk. No patches applied.
Sebastian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@gmx.net>.
I repeated the test from my original post.
Add/commit of a single file named "test".
server: stock 1.3.0 svnserve, Windows XP, NTFS
client: trunk build, Windows XP, NTFS
1. copy, read test, write .svn\tmp\text-base\test.svn-base.tmp (512b blocks)
stack: apr_file_copy <- svn_io_copy_file <- svn_wc_transmit_text_deltas
<- svn_client__do_commit
2. transmit, read .svn\tmp\text-base\test.svn-base (100kb blocks)
stack: svn_txdelta__vdelta <- compute_window <- svn_txdelta_next_window
<- svn_txdelta_send_txstream <- svn_wc_transmit_text_deltas <-
svn_client__do_commit
3. hashing, read .svn\tmp\text-base\test.svn-base (512b blocks)
stack: svn_io_file_checksum <- svn_wc_transmit_text_deltas <-
svn_client__do_commit
4. hashing, read .svn\tmp\text-base\test.svn-base (512b blocks)
stack: svn_io_file_checksum <- svn_wc_process_committed2
5. compare, read test and .svn\tmp\text-base\test.svn-base simultaneous
(512b blocks)
stack: contents_identical_p <- svn_io_files_contents_same_p <-
compare_and_verify <- svn_wc__versioned_file_modcheck <-
log_do_committed <- start_handler <- expat_start_handler <- doContent <-
contentProcessor <- XML_ParseBuffer <- XML_Parse <- svn_xml_parse <-
run_log <- svn_wc__run_log <- svn_wc_process_committed2
Most obvious problem is that test.svn-base gets hashed two times. Is the
comparison (point 5) necessary?
Sebastian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Philip Martin <ph...@codematters.co.uk>.
"Peter N. Lundblad" <pe...@famlundblad.se> writes:
> If using DAV, then the whole request will be put in a temporary file
> and then read back again and sent to the server.
I don't think that causes any additional reading of .svn-base files.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Philip Martin writes:
> Sebastian Tusk <se...@gmx.net> writes:
>
> As I understand it the commit will:
>
> - copy (i.e. read/write) the file to a temporary text-base so that it
> won't subsequently change during the rest of the commit.
>
> - read the temporary text-base to get a checksum. I suppose we could
> combine this with the copy above but that's quite complex and would
> involve moddifying apr_file_copy (or not using it).
>
> - read the temporary text-base to calculate the delta.
If using DAV, then the whole request will be put in a temporary file
and then read back again and sent to the server.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Philip Martin <ph...@codematters.co.uk>.
Sebastian Tusk <se...@gmx.net> writes:
> The file to commit is named test. The function performed is guessed.
>
> activity function? time
> ------
> read 512b blocks from "test"
> write 512b blocks to "test.svn_base.tmp" COPY? 3m10s
> ------
> read 102400b blocks from "test.svn-base"
> write 4096b blocks to "temp.tmp"
> occasionally other reads from "test.svn-base" COPY? DIFF? 2m23s
> ------
> read 512b blocks from test.svn-base HASHING? 1m47s
> ------
> read 4096b blocks from "tempfile" TRANSFER TO SERVER 7m8s
> ------
> read 512b blocks from "test.svn-base" HASHING? 1m16s
> ------
> read 512b blocks from "test"
> read 512b blocks from "test.svn-base" COMPARE? 7m46s
> ------
> read 512b blocks from "test.svn-base"
> write 512b blocks to "test.tmp.tmp"
> occasionally other reads from "test.svn-base" COPY? 3m12s
> ------
> read 512b blocks from "test"
> read 512b blocks from "test.tmp" COMPARE? 7m42s
>
> That are all activities that consume significant time. All reads and
> writes are over the full 680MB file. Only during TRANSFER TO SERVER
> there is any noticible server activity. Alltogether the commit command
> uses more than 13 million read or write operations.
>
> Are all this activities necessary?
As I understand it the commit will:
- copy (i.e. read/write) the file to a temporary text-base so that it
won't subsequently change during the rest of the commit.
- read the temporary text-base to get a checksum. I suppose we could
combine this with the copy above but that's quite complex and would
involve moddifying apr_file_copy (or not using it).
- read the temporary text-base to calculate the delta.
Once the commit has succeeded the post commit processing will:
- read the temporary text-base to calculate the checksum. There is a
comment about whether we could reuse the previous checksum. Since
the file is in the .svn area this should be possible.
- read the working and temporary text-base to determine whether the
working file has changed. This is necessary to determine whether
the text-time in the entries file should be set to match the working
file.
I make that 5 reads but you seem to be getting more, I don't know why.
> Especially those after the transfer
> to the server is finished. Is it necessary to access two files in
> parallel in such small blocks? Using larger read/write blocks might
> improve the performance significant without to much hassle.
Using BUFSIZ is probably wrong, particularly if it is 512.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Branko Čibej <br...@xbc.nu>.
Sebastian Tusk wrote:
> I did some tests with a repository that contains a single file sized
> 686 MB. I used the svn client version 1.3.0 on a windows machine.
>
> svn commit 1587s
> svn export 136s
> svn co 395s
>
> To me this figures appeared way to high. It seems that most of the
> time is used up in disk activities so I had took a look at that.
You don't say if your file has svn:eol-style or svn:keywords properties.
I'm assuming it does.
>
> svn commit
> The file to commit is named test. The function performed is guessed.
>
> activity function? time
> ------
> read 512b blocks from "test"
> write 512b blocks to "test.svn_base.tmp" COPY? 3m10s
This is probably "detranslate". We should be skipping this part if the
file doesn't have svn:eol-style or svn::keywords properties.
With the change on trunk that replaces detranslation to a temporary file
with a detranslating stream, these writes should go away.
> ------
> read 102400b blocks from "test.svn-base"
> write 4096b blocks to "temp.tmp"
> occasionally other reads from "test.svn-base" COPY? DIFF? 2m23s
"Delta" on the detranslated file. Again, we could probably get rid of
the read if we simply use a detranslating stream.
> ------
> read 512b blocks from test.svn-base HASHING? 1m47s
Possibly hashing, yes... dunno.
> ------
> read 4096b blocks from "tempfile" TRANSFER TO SERVER 7m8s
> ------
> read 512b blocks from "test.svn-base" HASHING? 1m16s
Hm. If that's hashing again, it should go away.
> ------
> read 512b blocks from "test"
> read 512b blocks from "test.svn-base" COMPARE? 7m46s
> ------
> read 512b blocks from "test.svn-base"
> write 512b blocks to "test.tmp.tmp"
> occasionally other reads from "test.svn-base" COPY? 3m12s
> ------
> read 512b blocks from "test"
> read 512b blocks from "test.tmp" COMPARE? 7m42s
I don't understand these last entries.
>
> That are all activities that consume significant time. All reads and
> writes are over the full 680MB file. Only during TRANSFER TO SERVER
> there is any noticible server activity. Alltogether the commit command
> uses more than 13 million read or write operations.
>
> Are all this activities necessary? Especially those after the transfer
> to the server is finished. Is it necessary to access two files in
> parallel in such small blocks? Using larger read/write blocks might
> improve the performance significant without to much hassle.
As I said, we could probably get rid of some of the reads and writes
(and I believe we have done so, on trunk). The fact that we read such
small blocks is definitely a bad thing.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by kf...@collab.net.
kfogel@collab.net writes:
> Julian Foad <ju...@btopenworld.com> writes:
> > Branko Čibej wrote:
> > > How about this one, as a stopgap:
> > > Index: subversion/libsvn_subr/io.c
> > > ===================================================================
> > > --- subversion/libsvn_subr/io.c (revision 18760)
> > > +++ subversion/libsvn_subr/io.c (working copy)
> > > @@ -988,7 +988,7 @@
> > > struct apr_md5_ctx_t context;
> > > apr_file_t *f = NULL;
> > > svn_error_t *err;
> > > - char buf[BUFSIZ]; /* What's a good size for a read chunk? */
> > > + char *buf = apr_palloc(pool, SVN__STREAM_CHUNK_SIZE);
> > > apr_size_t len;
> > > /* ### The apr_md5 functions return apr_status_t, but they only
> > > @@ -999,12 +999,12 @@
> > > SVN_ERR(svn_io_file_open(&f, file, APR_READ, APR_OS_DEFAULT,
> > > pool));
> > > - len = sizeof(buf);
> > > + len = SVN__STREAM_CHUNK_SIZE;
> > > err = svn_io_file_read(f, buf, &len, pool);
> > > while (! err)
> > > {
> > > apr_md5_update(&context, buf, len);
> > > - len = sizeof(buf);
> > > + len = SVN__STREAM_CHUNK_SIZE;
> > > err = svn_io_file_read(f, buf, &len, pool);
> > > };
> >
> > +1, except don't just change this function but also all the other
> > places we use it: that's two other functions in io.c, and also:
> > subversion/libsvn_wc/entries.c:read_entries(),
> > subversion/libsvn_wc/log.c:run_log(),
> > subversion/svnlook/main.c:do_cat().
>
> Why would we ever not use sizeof() when calculating len, though? It's
> always right. It can never be wrong. To the argument that we would
> never change the buffer's size, I point to this patch as
> counter-evidence :-).
>
> +1 on moving from BUFSIZ to SVN__STREAM_CHUNK_SIZE, -0.9 on writing
> out SVN__STREAM_CHUNK_SIZE instead of just using sizeof().
D'oh, my apologies. I read too quickly and missed the fact that we're
going from static allocation to pool allocation.
Stepping away from the computer,
-Karl
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 09 Mar 2006 17:09:35 -0600, kfogel@collab.net <kf...@collab.net> wrote:
> Why would we ever not use sizeof() when calculating len, though? It's
> always right. It can never be wrong. To the argument that we would
> never change the buffer's size, I point to this patch as
> counter-evidence :-).
sizeof is wrong in this case because it would be sizeof the pointer,
not sizeof the array, since it switches to dynamically allocating the
buffer.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by kf...@collab.net.
Julian Foad <ju...@btopenworld.com> writes:
> Branko Čibej wrote:
> > How about this one, as a stopgap:
> > Index: subversion/libsvn_subr/io.c
> > ===================================================================
> > --- subversion/libsvn_subr/io.c (revision 18760)
> > +++ subversion/libsvn_subr/io.c (working copy)
> > @@ -988,7 +988,7 @@
> > struct apr_md5_ctx_t context;
> > apr_file_t *f = NULL;
> > svn_error_t *err;
> > - char buf[BUFSIZ]; /* What's a good size for a read chunk? */
> > + char *buf = apr_palloc(pool, SVN__STREAM_CHUNK_SIZE);
> > apr_size_t len;
> > /* ### The apr_md5 functions return apr_status_t, but they only
> > @@ -999,12 +999,12 @@
> > SVN_ERR(svn_io_file_open(&f, file, APR_READ, APR_OS_DEFAULT,
> > pool));
> > - len = sizeof(buf);
> > + len = SVN__STREAM_CHUNK_SIZE;
> > err = svn_io_file_read(f, buf, &len, pool);
> > while (! err)
> > {
> > apr_md5_update(&context, buf, len);
> > - len = sizeof(buf);
> > + len = SVN__STREAM_CHUNK_SIZE;
> > err = svn_io_file_read(f, buf, &len, pool);
> > };
>
> +1, except don't just change this function but also all the other
> places we use it: that's two other functions in io.c, and also:
> subversion/libsvn_wc/entries.c:read_entries(),
> subversion/libsvn_wc/log.c:run_log(),
> subversion/svnlook/main.c:do_cat().
Why would we ever not use sizeof() when calculating len, though? It's
always right. It can never be wrong. To the argument that we would
never change the buffer's size, I point to this patch as
counter-evidence :-).
+1 on moving from BUFSIZ to SVN__STREAM_CHUNK_SIZE, -0.9 on writing
out SVN__STREAM_CHUNK_SIZE instead of just using sizeof().
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Julian Foad <ju...@btopenworld.com>.
Branko Čibej wrote:
>
> How about this one, as a stopgap:
>
> Index: subversion/libsvn_subr/io.c
> ===================================================================
> --- subversion/libsvn_subr/io.c (revision 18760)
> +++ subversion/libsvn_subr/io.c (working copy)
> @@ -988,7 +988,7 @@
> struct apr_md5_ctx_t context;
> apr_file_t *f = NULL;
> svn_error_t *err;
> - char buf[BUFSIZ]; /* What's a good size for a read chunk? */
> + char *buf = apr_palloc(pool, SVN__STREAM_CHUNK_SIZE);
> apr_size_t len;
>
> /* ### The apr_md5 functions return apr_status_t, but they only
> @@ -999,12 +999,12 @@
>
> SVN_ERR(svn_io_file_open(&f, file, APR_READ, APR_OS_DEFAULT, pool));
>
> - len = sizeof(buf);
> + len = SVN__STREAM_CHUNK_SIZE;
> err = svn_io_file_read(f, buf, &len, pool);
> while (! err)
> {
> apr_md5_update(&context, buf, len);
> - len = sizeof(buf);
> + len = SVN__STREAM_CHUNK_SIZE;
> err = svn_io_file_read(f, buf, &len, pool);
> };
+1, except don't just change this function but also all the other places we use
it: that's two other functions in io.c, and also:
subversion/libsvn_wc/entries.c:read_entries(),
subversion/libsvn_wc/log.c:run_log(),
subversion/svnlook/main.c:do_cat().
At a quick look, it seems right to change all of these to use a pool-allocated
buffer of SVN__STREAM_CHUNK_SIZE bytes.
- Julian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Branko Čibej <br...@xbc.nu>.
Peter N. Lundblad wrote:
> Sebastian Tusk writes:
> > I did some tests with a repository that contains a single file sized 686
> > MB. I used the svn client version 1.3.0 on a windows machine.
> >
> [...]
> > svn commit
> > The file to commit is named test. The function performed is guessed.
> >
> > activity function? time
> > ------
> > read 512b blocks from "test"
> > write 512b blocks to "test.svn_base.tmp" COPY? 3m10s
>
> Copy the file to a temporary text base. 512 byte blocks are used on
> Windows because APRs file copying routine uses BUFSIZ which is far too
> small on Windows.
Ah, yes. In fact, APR doesn't even have a Windows-specific
implementation of apr_file_copy (which should use the hopefully optimal
CopyFile API).
> See
> <http://svn.haxx.se/dev/archive-2004-11/0123.shtml> for some
> discussion. I think we should have our own copy function (as a
> short-term solution) if it improves performance dramatically on some platforms.
>
Probably most, not just some platforms. Of course it depends on the
filesystem and other considerations, but BUFSIZ is intended for use by
the stdio library, not raw file IO.
> There have been some work in this area for 1.4. For example, temporary
> file creation for MD5 hashing has been eliminated in some cases. It
> would be nice to see results using a trunk build. And it would be nice
> to see the performance changes when increasing BUFSIZ. (Oh, and it
> seems that svn_io_file_checksum also uses BUFGSIZ).
>
Ouch. We should be using SVN__STREAM_CHUNK_SIZE there.
> Would you be willing to try a trunk build with the above
> modifications? I would be willing to review patches...
>
How about this one, as a stopgap:
Index: subversion/libsvn_subr/io.c
===================================================================
--- subversion/libsvn_subr/io.c (revision 18760)
+++ subversion/libsvn_subr/io.c (working copy)
@@ -988,7 +988,7 @@
struct apr_md5_ctx_t context;
apr_file_t *f = NULL;
svn_error_t *err;
- char buf[BUFSIZ]; /* What's a good size for a read chunk? */
+ char *buf = apr_palloc(pool, SVN__STREAM_CHUNK_SIZE);
apr_size_t len;
/* ### The apr_md5 functions return apr_status_t, but they only
@@ -999,12 +999,12 @@
SVN_ERR(svn_io_file_open(&f, file, APR_READ, APR_OS_DEFAULT, pool));
- len = sizeof(buf);
+ len = SVN__STREAM_CHUNK_SIZE;
err = svn_io_file_read(f, buf, &len, pool);
while (! err)
{
apr_md5_update(&context, buf, len);
- len = sizeof(buf);
+ len = SVN__STREAM_CHUNK_SIZE;
err = svn_io_file_read(f, buf, &len, pool);
};
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@gmx.net>.
> > I do that if someone provides a windows build. I don't know when I would
> > find the time to build it myself. I guess it takes a while to setup the
> > build environment.
> >
> As I said before, I can send compiled windows build if you want. I can
> do it tomorow. Is your email server accepts big attachements (10 mb)?
Yes, attachments of this size should be no problem. I will gladly check out
your builds.
--
Bis zu 70% Ihrer Onlinekosten sparen: GMX SmartSurfer!
Kostenlos downloaden: http://www.gmx.net/de/go/smartsurfer
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Ivan Zhakov <ch...@gmail.com>.
On 3/9/06, Sebastian Tusk <se...@gmx.net> wrote:
> Peter N. Lundblad wrote:
> > Would you be willing to try a trunk build with the above
> > modifications? I would be willing to review patches...
>
> I do that if someone provides a windows build. I don't know when I would
> find the time to build it myself. I guess it takes a while to setup the
> build environment.
>
As I said before, I can send compiled windows build if you want. I can
do it tomorow. Is your email server accepts big attachements (10 mb)?
--
Ivan Zhakov
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@gmx.net>.
Peter N. Lundblad wrote:
> Would you be willing to try a trunk build with the above
> modifications? I would be willing to review patches...
I do that if someone provides a windows build. I don't know when I would
find the time to build it myself. I guess it takes a while to setup the
build environment.
Sebastian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by Sebastian Tusk <se...@silver-style.com>.
Peter N. Lundblad wrote:
> Would you be willing to try a trunk build with the above
> modifications? I would be willing to review patches...
I do that if someone provides a windows build. I don't know when I would
find the time to build it myself. I guess it takes a while to setup the
build environment.
Sebastian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: export, checkout, commit performance
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Sebastian Tusk writes:
> I did some tests with a repository that contains a single file sized 686
> MB. I used the svn client version 1.3.0 on a windows machine.
>
[...]
> svn commit
> The file to commit is named test. The function performed is guessed.
>
> activity function? time
> ------
> read 512b blocks from "test"
> write 512b blocks to "test.svn_base.tmp" COPY? 3m10s
Copy the file to a temporary text base. 512 byte blocks are used on
Windows because APRs file copying routine uses BUFSIZ which is far too
small on Windows. See
<http://svn.haxx.se/dev/archive-2004-11/0123.shtml> for some
discussion. I think we should have our own copy function (as a
short-term solution) if it improves performance dramatically on some platforms.
There have been some work in this area for 1.4. For example, temporary
file creation for MD5 hashing has been eliminated in some cases. It
would be nice to see results using a trunk build. And it would be nice
to see the performance changes when increasing BUFSIZ. (Oh, and it
seems that svn_io_file_checksum also uses BUFGSIZ).
Would you be willing to try a trunk build with the above
modifications? I would be willing to review patches...
Thanks for the analyzes so far,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org