You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Michael Wood <mw...@its.uct.ac.za> on 2003/04/11 10:53:58 UTC
Re: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
On Thu, Apr 10, 2003 at 04:35:46PM -0000, issues@subversion.tigris.org wrote:
> http://subversion.tigris.org/issues/show_bug.cgi?id=1240
>
[snip]
> + ------- Additional Comments From kfogel@tigris.org 2003-04-10 09:35 PDT -------
> + Mike Pilato points out that the problem may not be the one-byte reads,
> + since that I/O is presumably buffered. So the first thing is to do
^^^^^^^^^^
Is it though? Haven't checked. This assumption just jumped out at me
:)
--
Michael Wood <mw...@its.uct.ac.za>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
Posted by Robert Pluim <rp...@bigfoot.com>.
Karl Fogel writes:
> Michael Wood <mw...@its.uct.ac.za> writes:
> > > + Mike Pilato points out that the problem may not be the one-byte reads,
> > > + since that I/O is presumably buffered. So the first thing is to do
> > ^^^^^^^^^^
> >
> > Is it though? Haven't checked. This assumption just jumped out at me
> > :)
>
> Yup, that wasn't an assertion, it was merely a note *not* to blindly
> assume that I/O is the problem.
Two data points:
in svn_subst_copy_and_translate, the source file used for translation
is opened as follows:
SVN_ERR (svn_io_file_open (&s, src, APR_READ | APR_BUFFERED,
APR_OS_DEFAULT, pool));
which tells me that it _should_ be buffered, right?
Secondly, on my FreeBSD box, checking out the subversion source over
http gives the following profile info for svn_subst_translate_stream
called/total parents
index %time self descendents called+self name index
called/total children
0.48 4.18 730/730 svn_subst_copy_and_translate [6]
[7] 26.0 0.48 4.18 730 svn_subst_translate_stream [7]
0.25 1.98 9630507/9925261 translate_write [13]
0.22 1.65 9968018/9968018 svn_stream_read [16]
0.01 0.07 294754/294754 translate_newline [102]
0.00 0.00 1460/3898 svn_stream_close [743]
0.00 0.00 770/770 translate_keyword [758]
translate_write doesn't really do much, but it's called an awful lot.
I replaced it with a shameful hack which just buffers all the bytes
and calls svn_stream_write once at the end of the file, and got:
0.58 2.02 731/731 svn_subst_copy_and_translate [11]
[12] 15.7 0.58 2.02 731 svn_subst_translate_stream [12]
0.22 1.63 9975019/9975019 svn_stream_read [15]
0.17 0.00 9637151/9932110 translate_write [73]
0.00 0.01 294959/294959 translate_newline [276]
0.00 0.00 1462/3903 svn_stream_close [390]
0.00 0.00 731/13248 svn_stream_write [215]
0.00 0.00 770/770 translate_keyword [670]
Of course, that wouldn't work as a general solution, but we could
consider having a buffer in svn_subst_translate_stream that we flush
to disk every n characters, to avoid having to descend the call chain
into the depths of apr for every byte. Something similar could be
done for svn_stream_read, but that's more work.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Michael Wood <mw...@its.uct.ac.za> writes:
> > + Mike Pilato points out that the problem may not be the one-byte reads,
> > + since that I/O is presumably buffered. So the first thing is to do
> ^^^^^^^^^^
>
> Is it though? Haven't checked. This assumption just jumped out at me
> :)
Yup, that wasn't an assertion, it was merely a note *not* to blindly
assume that I/O is the problem.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
Posted by Greg Stein <gs...@lyra.org>.
On Fri, Apr 11, 2003 at 08:40:33AM -0500, Karl Fogel wrote:
> "Tim Kemp" <su...@kempy.co.uk> writes:
> > Hmmmm, 10% of checkout time is spent in svn_subst_copy_and_translate.
> > With 10 million calls all the way down to apr_file_read and
> > apr_file_write.
>
> That's a very bizarre result.
>
> Ben's export editor, before he added the translation calls, was 4
> times faster than a regular checkout. When he added translation, it
> went back to being about the same amount of time as a checkout. (I
> will check this to make sure, but I think adding translation boils
> down to svn_subst_copy_and_translate() being called for each file.)
>
> And yet, you're saying that only 10% of checkout time is spent in
> svn_subst_copy_and_translate().
>
> At least one of these sets of numbers is misleading... :-)
Has Ben re-verified his numbers? Let's make sure that the numbers
we're all scurrying about for, are actually valid.
Regardless... 10% of checkout time is significant, thus is a fine
candidate for examination.
Cheers,
-g
--
Greg Stein, http://www.lyra.org/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
Posted by Robert Pluim <rp...@bigfoot.com>.
Tim Kemp writes:
> > > 10% of cpu time, right? That doesn't take into account
> > time waiting
> > > for disk, network etc.
> >
> > Oh, that would explain it. Who cares about CPU time? I live
> > in a wall clock world :-).
> >
> Errr, this is real time, time taken includes waits for file system/dav
> etc.
>
Ahh, ok. That wasn't clear to me.
> This is a remote checkout for the entire trunk, 6 mins total, 5 mins of
> that downloading the files!!!, 1 min in svn_subs_translate_stream
Right. I'm building an instrumented svn here right now to see if I
get similar numbers on FreeBSD. <waiting...> Hmm, looks like
svn.collab.net isn't very happy, I'll try again later tonight.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
Posted by Tim Kemp <su...@kempy.co.uk>.
> > 10% of cpu time, right? That doesn't take into account
> time waiting
> > for disk, network etc.
>
> Oh, that would explain it. Who cares about CPU time? I live
> in a wall clock world :-).
>
Errr, this is real time, time taken includes waits for file system/dav
etc.
This is a remote checkout for the entire trunk, 6 mins total, 5 mins of
that downloading the files!!!, 1 min in svn_subs_translate_stream
Tim
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
Robert Pluim <rp...@bigfoot.com> writes:
> 10% of cpu time, right? That doesn't take into account time waiting
> for disk, network etc.
Oh, that would explain it. Who cares about CPU time? I live in a
wall clock world :-).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
Posted by Robert Pluim <rp...@bigfoot.com>.
Karl Fogel writes:
> "Tim Kemp" <su...@kempy.co.uk> writes:
> > Hmmmm, 10% of checkout time is spent in svn_subst_copy_and_translate.
> > With 10 million calls all the way down to apr_file_read and
> > apr_file_write.
>
> That's a very bizarre result.
>
> Ben's export editor, before he added the translation calls, was 4
> times faster than a regular checkout. When he added translation, it
> went back to being about the same amount of time as a checkout. (I
> will check this to make sure, but I think adding translation boils
> down to svn_subst_copy_and_translate() being called for each file.)
>
> And yet, you're saying that only 10% of checkout time is spent in
> svn_subst_copy_and_translate().
>
10% of cpu time, right? That doesn't take into account time waiting
for disk, network etc.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
Posted by Karl Fogel <kf...@newton.ch.collab.net>.
"Tim Kemp" <su...@kempy.co.uk> writes:
> Hmmmm, 10% of checkout time is spent in svn_subst_copy_and_translate.
> With 10 million calls all the way down to apr_file_read and
> apr_file_write.
That's a very bizarre result.
Ben's export editor, before he added the translation calls, was 4
times faster than a regular checkout. When he added translation, it
went back to being about the same amount of time as a checkout. (I
will check this to make sure, but I think adding translation boils
down to svn_subst_copy_and_translate() being called for each file.)
And yet, you're saying that only 10% of checkout time is spent in
svn_subst_copy_and_translate().
At least one of these sets of numbers is misleading... :-)
-K
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: [Issue 1240] Changed - svn_subst_translate_stream() should read bigger chunks
Posted by Tim Kemp <su...@kempy.co.uk>.
Just to add some numbers to this (I'm not sure if this is the right
place, but, oh well)
On checking out the subversion tree over a 1Mbit cable line,
Name, Visits, % of root time, Time (secs), Avg Time, Child as % of
S+C Time
-svn_subst_copy_and_translate, 978, 9.4%, 63.729745, 0.065163, 100.0%,
0, 0, 0
-svn_subst_translate_stream, 730, 5.3%, 62.820510, 0.086055, 91.3%,
0, 0, 0
+translate_write, 9620166, 2.5%, 30.180774, 0.000003, 84.6%, 0, 0,
0
-svn_stream_read, 9957327, 2.2%, 26.104806, 0.000003, 81.9%, 0, 0,
0
-read_handler_apr, 9957327, 1.5%, 21.383350, 0.000002, 78.8%, 0,
0, 0
-apr_file_read_full, 9957327, 1.2%, 16.847580, 0.000002,
72.8%, 0, 0, 0
-apr_file_read, 9957327, 0.8%, 12.271248, 0.000001, 35.9%,
0, 0, 0
apr_thread_mutex_unlock, 9957327, 0.2%, 2.216958,
0.000000, 0.0%, 0, 0, 0
apr_thread_mutex_lock, 9957327, 0.1%, 2.151563, 0.000000,
0.0%, 0, 0, 0
read_with_timeout, 3544, 0.0%, 0.041370, 0.000012, 0.0%,
0, 0, 0
Hmmmm, 10% of checkout time is spent in svn_subst_copy_and_translate.
With 10 million calls all the way down to apr_file_read and
apr_file_write.
System is Win XP, Athlon 1800, 512 Mb RAM, Fast IDE HD, Subversion R5605
Hope this helps
Tim
> -----Original Message-----
> From: Michael Wood [mailto:mwood@its.uct.ac.za]
> Sent: 11 April 2003 11:54
> To: dev@subversion.tigris.org
> Cc: issues@subversion.tigris.org
> Subject: Re: [Issue 1240] Changed -
> svn_subst_translate_stream() should read bigger chunks
>
>
> On Thu, Apr 10, 2003 at 04:35:46PM -0000,
> issues@subversion.tigris.org wrote:
> > http://subversion.tigris.org/issues/show_bug.cgi?id=1240
> >
> [snip]
> > + ------- Additional Comments From kfogel@tigris.org
> 2003-04-10 09:35
> > + PDT ------- Mike Pilato points out that the problem may
> not be the
> > + one-byte reads, since that I/O is presumably buffered.
> So the first
> > + thing is to do
> ^^^^^^^^^^
>
> Is it though? Haven't checked. This assumption just jumped out at me
> :)
>
> --
> Michael Wood <mw...@its.uct.ac.za>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org