You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "Peter N. Lundblad" <pe...@famlundblad.se> on 2006/04/06 22:00:21 UTC
[PROPOSAL] Drop XML from .svn/entries
Hi,
"Just For Fun", I experimented with replacing our current XML-based
.svn/entries syntax with a line-based, |-separated syntax. It
actually turns out that there is some overhead associated with using
XML, but it is not extreme overhead.
To get some idea how much overhead there actually is, I tested
something that might be considered as a "typical work cycle". I used
the following one-liner:
time (svn st; (sync&); sleep 10; svn up; (sync&); sleep 10; svn st; \
(sync&); sleep 10; svn ci)
The point is to do some common operations in a row, but get the disk
flushed in between, since you typically will not do the operations
this quickly (you look at the output, do some editing, etc). Then I
subtracted the times for the sleeps and compared the wallclock times
for these operations. (As usual, I tried with the GCC tree on my
Celereon 17. GHz using LInux).
The result is that this row of operation requires 19% less wallclock
time and 35% less CPU time. NOte that I'm not an expert in
performance measuring, so feel free to just call this method bullshit
or whatever:-)
ON another note, the size of all the entries files in the GCC tree
decreased from 8.2 to 4.2 megabytes, which probably explains much of
the time savings. (The size is not a big deal, since where we
currently consume peoples' disk space is in the textbase files.)
So, I propose to replace our current XML entries files with some
variant of this new format. Below is three lines from an entries file
to show how it might look:
|dir|2|file:///home/pl/svntest/gcc/repo|file:///home/pl/svntest/gcc/repo||||2006-04-06T20:31:40.024395Z|2|pl|t||svn:special svn:externals svn:needs-lock||||||||||||b4f160ba-c4e8-11da-bcd8-65faf9067726|
config-ml.in|file|||||2006-04-05T21:35:10.000000Z|e09724e0a7725cc08129db9293a35df9|2006-04-05T21:20:33.744115Z|1|pl|
libgomp|dir|
...
Each entry is on a separated line and each field is terminated by a |
character. Empty fields may be omitted provided all following fields
are also empty. Absent fields are represented by empty fields (except
for the entry name, where an empty field means the thisdir entry).
Boolean values are represent by "t" (for true" or an empty field for
false. IN the future, we can add new fields at the end of each
record. There will be an escaping mechanism for control characters
(think newlines in lock comments) and the | separator.
Since we are not used any of the things XML provides such as
structure, the only drawback I see to making this change is a slight
degradation in readability. But hey, computers will read this file
much more often than humans, and we could easily write a
pretty-printing script for debugging purposes. Another thing that
might be considered a drawback is that we will need to maintain botht
the old and new entries parsing code for compatibility. The parsing
code for this new format, however, is pretty straightforward, so I
don't think this is a problem.
If we want this, we should do it before 1.4, since we already have WC
format upgrades.
Opinions?
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Ivan Zhakov <ch...@gmail.com>.
On 4/18/06, John Szakmeister <jo...@szakmeister.net> wrote:
> On Tuesday 18 April 2006 00:15, Ivan Zhakov wrote:
> > On 4/18/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > > Garrett Rooney writes:
> > > > On 4/17/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > > > > Garrett Rooney writes:
> > > >
> > > > I was assuming that each entry had a fixed number of fields or
> > > > something like that. If not, I imagine we could include some specific
> > > > character as the end of entry marker...
> > >
> > > After some discussion with rooneg on IRC, I'm going to change the format
> > > so that newlines separate fields instead of pipes and the sequence "\f\n"
> > > (control-l newline) will terminate each record.
> >
> > What is \f means? Did you mean "\r\n"?
>
> '\f' is the form-feed character (it advances to the next page when printing).
>
> >
> > > Can everybody live with this change?
> >
> > -0.5, I cannot. IMHO such format could be broken by many editors. I
> > often tweak entries file for debugging.
>
> Doubtful... they're present in the source code right now, and your editor
> seems to be doing fine. :-) Take a look just below the copyright header, and
> you'll find one. Just below #includes, you'll find another. Most editors
> I've come across seem to work fine with the form-feed character.
Oh, sorry. Ok, I am removing my vote, if it's new page separator. I've
investigated.
--
Ivan Zhakov
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Vincent Lefevre <vi...@vinc17.org>.
On 2006-04-18 22:19:51 +0200, Peter N. Lundblad wrote:
> Well... I find some of your examples above somewhat contrived.
> If you want to document this in HTML with examples, you'll need to
> find a notation that loos good in a web browser. I can't see how that
> could reasonably be a factor when deciding upon the file format. We
> are talking about an internal data format here, not a format that
> users normally will have to deal with.
I agree, but if one can avoid control characters without drawbacks,
that's better.
> What would you propose as field/record separators?
I like mark benedetto king's proposition very much, i.e. something like:
Nfoo.xml
R19127
T2006-04-14T04:13:41.000000Z
Ajrandom
Kfile
...more attributes...
X
IMHO, it is more readable, less prone to bugs, more flexible, more
extensible, and with slightly easier corruption detection.
--
Vincent Lefèvre <vi...@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Vincent Lefevre writes:
> On 2006-04-18 03:38:13 -0400, John Szakmeister wrote:
> > Doubtful... they're present in the source code right now, and your
> > editor seems to be doing fine. :-) Take a look just below the
> > copyright header, and you'll find one. Just below #includes, you'll
> > find another. Most editors I've come across seem to work fine with
> > the form-feed character.
>
> What if you want to describe the file format on the web page and give
> an example? What if you want to do a copy-paste from a text terminal?
> What if you want to include a part of the file in a mail (e.g. for a
> bug report)?
>
> Also, the control character needs to be displayed by the editor under
> some form, and it could be confused with normal characters.
>
> IMHO, for all these reasons, it would be better to avoir control
> characters (except for the end of lines, of course) and trailing
> spaces.
>
Well... I find some of your examples above somewhat contrived.
If you want to document this in HTML with examples, you'll need to
find a notation that loos good in a web browser. I can't see how that
could reasonably be a factor when deciding upon the file format. We
are talking about an internal data format here, not a format that
users normally will have to deal with.
What would you propose as field/record separators?
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Vincent Lefevre <vi...@vinc17.org>.
On 2006-04-18 03:38:13 -0400, John Szakmeister wrote:
> Doubtful... they're present in the source code right now, and your
> editor seems to be doing fine. :-) Take a look just below the
> copyright header, and you'll find one. Just below #includes, you'll
> find another. Most editors I've come across seem to work fine with
> the form-feed character.
What if you want to describe the file format on the web page and give
an example? What if you want to do a copy-paste from a text terminal?
What if you want to include a part of the file in a mail (e.g. for a
bug report)?
Also, the control character needs to be displayed by the editor under
some form, and it could be confused with normal characters.
IMHO, for all these reasons, it would be better to avoir control
characters (except for the end of lines, of course) and trailing
spaces.
--
Vincent Lefèvre <vi...@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by John Szakmeister <jo...@szakmeister.net>.
On Tuesday 18 April 2006 00:15, Ivan Zhakov wrote:
> On 4/18/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > Garrett Rooney writes:
> > > On 4/17/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > > > Garrett Rooney writes:
> > >
> > > I was assuming that each entry had a fixed number of fields or
> > > something like that. If not, I imagine we could include some specific
> > > character as the end of entry marker...
> >
> > After some discussion with rooneg on IRC, I'm going to change the format
> > so that newlines separate fields instead of pipes and the sequence "\f\n"
> > (control-l newline) will terminate each record.
>
> What is \f means? Did you mean "\r\n"?
'\f' is the form-feed character (it advances to the next page when printing).
>
> > Can everybody live with this change?
>
> -0.5, I cannot. IMHO such format could be broken by many editors. I
> often tweak entries file for debugging.
Doubtful... they're present in the source code right now, and your editor
seems to be doing fine. :-) Take a look just below the copyright header, and
you'll find one. Just below #includes, you'll find another. Most editors
I've come across seem to work fine with the form-feed character.
-John
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 4/17/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> After some discussion with rooneg on IRC, I'm going to change the format so
> that newlines separate fields instead of pipes and the sequence "\f\n"
> (control-l newline) will terminate each record.
>
> Can everybody live with this change?
+1. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Vincent Lefevre <vi...@vinc17.org>.
On 2006-04-18 18:05:31 +0200, Peter N. Lundblad wrote:
> If an editor munges, \x0c in a file, then stop using that editor.
This is not the only problem (see my other message about all the
problems). In general, avoiding control characters in file formats
would make bug report easier to write/understand, and ditto for
discussions.
> I choose to escape all control characters to avoid this and for
> simplicity.
So, the \ character needs to be escaped too. In this case, wouldn't
it be better to have a single \ in a line as the record terminator?
--
Vincent Lefèvre <vi...@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Vincent Lefevre writes:
> On 2006-04-17 23:25:56 +0200, Peter N. Lundblad wrote:
> > Greg Hudson writes:
> > > What if a field consists of just "\f"?
> >
> > Then it will be escaped like \x0c. All control characters are escaped
> > and likewise \.
>
> Do you mean that fields may contain control characters?
> Then how did you do that with XML? For instance,
>
> <?xml version="1.0"?>
> <test></test>
Yes, that's not wellformed XML 1.0, so we didn't allow control characters until
now, and we don't now either, since the things that could possibly
contain such beasts would be filenames and lock comments. None of
these may contain control characters today, among other things, for
compatibility with DAV and because we felt it wasn't worth the
complexity. But I don't want to impose XML's limitations on this file
format, so everything except NULs is allowed.
> Also, couldn't such control characters be munged by a text editor?
>
If an editor munges, \x0c in a file, then stop using that editor. I
choose to escape all control characters to avoid this and for
simplicity. Oh, and if your editor munges newline or form feed
(unescaped), then don't edit the entries file by that editor either.
Same if your editor inserts an UTF-8 BOM at the beginning of the file
or converts space to tabs or whatever. The entries file is made
read-only for a reason:-)
Best,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Vincent Lefevre <vi...@vinc17.org>.
On 2006-04-17 23:25:56 +0200, Peter N. Lundblad wrote:
> Greg Hudson writes:
> > What if a field consists of just "\f"?
>
> Then it will be escaped like \x0c. All control characters are escaped
> and likewise \.
Do you mean that fields may contain control characters?
Then how did you do that with XML? For instance,
<?xml version="1.0"?>
<test></test>
is incorrect (xmllint says: "parser error : xmlParseCharRef: invalid
xmlChar value 12").
Also, couldn't such control characters be munged by a text editor?
--
Vincent Lefèvre <vi...@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Greg Hudson writes:
> On Mon, 2006-04-17 at 22:35 +0200, Peter N. Lundblad wrote:
> > After some discussion with rooneg on IRC, I'm going to change the format so
> > that newlines separate fields instead of pipes and the sequence "\f\n"
> > (control-l newline) will terminate each record.
>
> What if a field consists of just "\f"?
Then it will be escaped like \x0c. All control characters are escaped
and likewise \.
> (To the person who suggested using the number fields: I assume we want
> the capability of adding fields in the future. So we can't rely on the
> number of fields being constant.)
Yeah, I explained that on IRC. For example, in 1.2, we added fields
for locking *without* bumping the WC format number. It meant that old
clients would silently drop the working copy lock information, but we
felt that was acceptable in this case. If wc-propcaching hadn't
involed changing the way working props are stored, we could have done
the same here to avoid a bump (there was already a bump, but you get
the point). We don't want to close this door for the future.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Vincent Lefevre <vi...@vinc17.org>.
On 2006-04-17 17:12:39 -0400, Greg Hudson wrote:
> What if a field consists of just "\f"?
>
> I can live with the idea, but instead of trying to make up a
> distinguished terminator field value, you might want to start each field
> with a field marker charater (say, a space) and then make the terminator
> line blank. It's some extra bytes, but it's more general.
Yes, but I'd avoid the space as a marker character, as for empty
fields, it could be removed by a text editor (for those who need
to fix things manually, do tests, debug and so on). Also, another
character would be more visible.
--
Vincent Lefèvre <vi...@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by mark benedetto king <mb...@lowlatency.com>.
On Mon, Apr 17, 2006 at 05:12:39PM -0400, Greg Hudson wrote:
> On Mon, 2006-04-17 at 22:35 +0200, Peter N. Lundblad wrote:
> > After some discussion with rooneg on IRC, I'm going to change the format so
> > that newlines separate fields instead of pipes and the sequence "\f\n"
> > (control-l newline) will terminate each record.
>
> What if a field consists of just "\f"?
>
> I can live with the idea, but instead of trying to make up a
> distinguished terminator field value, you might want to start each field
> with a field marker charater (say, a space) and then make the terminator
> line blank. It's some extra bytes, but it's more general.
>
> (To the person who suggested using the number fields: I assume we want
> the capability of adding fields in the future. So we can't rely on the
> number of fields being constant.)
>
I hate to muddy the water, but if we're going to do that, then
maybe it would make sense to do something like
Nfoo.xml
R19127
T2006-04-14T04:13:41.000000Z
Ajrandom
Kfile
...more attributes...
X
This would make the format easy-ish for both computers and humans
(especially COBOL programmers).
--ben
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Greg Hudson <gh...@MIT.EDU>.
On Mon, 2006-04-17 at 22:35 +0200, Peter N. Lundblad wrote:
> After some discussion with rooneg on IRC, I'm going to change the format so
> that newlines separate fields instead of pipes and the sequence "\f\n"
> (control-l newline) will terminate each record.
What if a field consists of just "\f"?
I can live with the idea, but instead of trying to make up a
distinguished terminator field value, you might want to start each field
with a field marker charater (say, a space) and then make the terminator
line blank. It's some extra bytes, but it's more general.
(To the person who suggested using the number fields: I assume we want
the capability of adding fields in the future. So we can't rely on the
number of fields being constant.)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Ivan Zhakov <ch...@gmail.com>.
On 4/18/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Garrett Rooney writes:
> > On 4/17/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > > Garrett Rooney writes:
> > I was assuming that each entry had a fixed number of fields or
> > something like that. If not, I imagine we could include some specific
> > character as the end of entry marker...
> >
>
> After some discussion with rooneg on IRC, I'm going to change the format so
> that newlines separate fields instead of pipes and the sequence "\f\n"
> (control-l newline) will terminate each record.
What is \f means? Did you mean "\r\n"?
>
> Can everybody live with this change?
>
-0.5, I cannot. IMHO such format could be broken by many editors. I
often tweak entries file for debugging.
--
Ivan Zhakov
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Garrett Rooney writes:
> On 4/17/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > Garrett Rooney writes:
> I was assuming that each entry had a fixed number of fields or
> something like that. If not, I imagine we could include some specific
> character as the end of entry marker...
>
After some discussion with rooneg on IRC, I'm going to change the format so
that newlines separate fields instead of pipes and the sequence "\f\n"
(control-l newline) will terminate each record.
Can everybody live with this change?
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 4/17/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Garrett Rooney writes:
> > On 4/16/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> >
> > > We could use spaces and require spaces to be escaped if that improves
> > > readability. (I guess my notion of readability is somewhat different
> > > than others'). I don't have a strong opinion here, but I think it is
> > > useful to use different separators for fields and records. Note that
> > > there is no known count of fields for the sake of extensibility.
> >
> > Spaces might help, because at least then when an editor wraps the line
> > it has a place to break... I still think newlines would work best
> > though.
> >
>
> How do you determine the end of a record when empty lines may be both empty
> fields and the end of the record?
I was assuming that each entry had a fixed number of fields or
something like that. If not, I imagine we could include some specific
character as the end of entry marker...
> (I think the best way to improve readability is a pretty-printing
> script or something like s/|/\n/, but I'm not really in love with the
> pipe characters either:-)
Oh sure, but I'd prefer if people could just look at the entries
directly without having to mess with them.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Garrett Rooney writes:
> On 4/16/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
>
> > We could use spaces and require spaces to be escaped if that improves
> > readability. (I guess my notion of readability is somewhat different
> > than others'). I don't have a strong opinion here, but I think it is
> > useful to use different separators for fields and records. Note that
> > there is no known count of fields for the sake of extensibility.
>
> Spaces might help, because at least then when an editor wraps the line
> it has a place to break... I still think newlines would work best
> though.
>
How do you determine the end of a record when empty lines may be both empty
fields and the end of the record?
(I think the best way to improve readability is a pretty-printing
script or something like s/|/\n/, but I'm not really in love with the
pipe characters either:-)
Best,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 4/16/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Well, I guess this is mostly a matter of taste. Personally, I find it more
> readable with one line than 30 per record. Zhakov suggested to switch
> to tab characters. I disliked that proposal because a tab might be
> confused to a space, but probably not often and never by a computer:-)
> But if you for some reason load the file in an editor that munges the
> tabs, your bets are off...
The main problem with one line is that the line will be extremely
long, which makes it wrap in 80 column editors/terminals/etc. That
makes it extremely hard to read in my opinion.
> We could use spaces and require spaces to be escaped if that improves
> readability. (I guess my notion of readability is somewhat different
> than others'). I don't have a strong opinion here, but I think it is
> useful to use different separators for fields and records. Note that
> there is no known count of fields for the sake of extensibility.
Spaces might help, because at least then when an editor wraps the line
it has a place to break... I still think newlines would work best
though.
> NOte that I (hopefully) improved readability by replacing boolean "t"
> with field names. Seeing "deleted" says far more than a "t" somewhere
> on the line. It increases the size of the file somewhat, but I don't
> think that's a probem performancewise.
I noticed that, and I think it will help a bit.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Michael Brouwer writes:
> Why not just use D for deleted since that's what svn status shows.
> Still only one char and it should be readable for anyone experienced
> using svn if you keep the the status output letter codes.
>
I think deleted is something other than what you think it is. In
particular, it is *not* the same as schedule delete. The reason I
changed this to use descriptive words is that, most often, the boolean
flags will be false and therefore absent from the entries file. The
same holds for the values of schedule; normal scheduling is marked by
an absent field (or empty if you prefer). The exception here is
has-props, which indicates that there are versioned properties
associated with an entry. This means that we've increased the entries
file size with some percents, but that's a price I find worth paying
for debuggability.
Best,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Michael Brouwer <mb...@gmail.com>.
Why not just use D for deleted since that's what svn status shows.
Still only one char and it should be readable for anyone experienced
using svn if you keep the the status output letter codes.
Michael
On 4/16/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Garrett Rooney writes:
> > On 4/6/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> >
> > > So, I propose to replace our current XML entries files with some
> > > variant of this new format. Below is three lines from an entries file
> > > to show how it might look:
> > >
> > > |dir|2|file:///home/pl/svntest/gcc/repo|file:///home/pl/svntest/gcc/repo||||2006-04-06T20:31:40.024395Z|2|pl|t||svn:special svn:externals svn:needs-lock||||||||||||b4f160ba-c4e8-11da-bcd8-65faf9067726|
> > > config-ml.in|file|||||2006-04-05T21:35:10.000000Z|e09724e0a7725cc08129db9293a35df9|2006-04-05T21:20:33.744115Z|1|pl|
> > > libgomp|dir|
> > > ...
> > >
> > Why not switch from pipe delimited to newline delimited fields? One
> > newline would delimit fields in an entry, a line with just a newline
> > on it would indicate a new entry. I suppose we might need some way to
> > indicate an empty field, but that should be easy enough, or heck, you
> > could even go based on the known number of fields in an entry, using
> > the blank line between entries as a sanity check. It would result in
> > far more easily readable entries files, and the size would be
> > identical...
> >
>
> Well, I guess this is mostly a matter of taste. Personally, I find it more
> readable with one line than 30 per record. Zhakov suggested to switch
> to tab characters. I disliked that proposal because a tab might be
> confused to a space, but probably not often and never by a computer:-)
> But if you for some reason load the file in an editor that munges the
> tabs, your bets are off...
>
> We could use spaces and require spaces to be escaped if that improves
> readability. (I guess my notion of readability is somewhat different
> than others'). I don't have a strong opinion here, but I think it is
> useful to use different separators for fields and records. Note that
> there is no known count of fields for the sake of extensibility.
>
> NOte that I (hopefully) improved readability by replacing boolean "t"
> with field names. Seeing "deleted" says far more than a "t" somewhere
> on the line. It increases the size of the file somewhat, but I don't
> think that's a probem performancewise.
>
> Night,
> //Peter
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Garrett Rooney writes:
> On 4/6/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
>
> > So, I propose to replace our current XML entries files with some
> > variant of this new format. Below is three lines from an entries file
> > to show how it might look:
> >
> > |dir|2|file:///home/pl/svntest/gcc/repo|file:///home/pl/svntest/gcc/repo||||2006-04-06T20:31:40.024395Z|2|pl|t||svn:special svn:externals svn:needs-lock||||||||||||b4f160ba-c4e8-11da-bcd8-65faf9067726|
> > config-ml.in|file|||||2006-04-05T21:35:10.000000Z|e09724e0a7725cc08129db9293a35df9|2006-04-05T21:20:33.744115Z|1|pl|
> > libgomp|dir|
> > ...
> >
> Why not switch from pipe delimited to newline delimited fields? One
> newline would delimit fields in an entry, a line with just a newline
> on it would indicate a new entry. I suppose we might need some way to
> indicate an empty field, but that should be easy enough, or heck, you
> could even go based on the known number of fields in an entry, using
> the blank line between entries as a sanity check. It would result in
> far more easily readable entries files, and the size would be
> identical...
>
Well, I guess this is mostly a matter of taste. Personally, I find it more
readable with one line than 30 per record. Zhakov suggested to switch
to tab characters. I disliked that proposal because a tab might be
confused to a space, but probably not often and never by a computer:-)
But if you for some reason load the file in an editor that munges the
tabs, your bets are off...
We could use spaces and require spaces to be escaped if that improves
readability. (I guess my notion of readability is somewhat different
than others'). I don't have a strong opinion here, but I think it is
useful to use different separators for fields and records. Note that
there is no known count of fields for the sake of extensibility.
NOte that I (hopefully) improved readability by replacing boolean "t"
with field names. Seeing "deleted" says far more than a "t" somewhere
on the line. It increases the size of the file somewhat, but I don't
think that's a probem performancewise.
Night,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 4/6/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> So, I propose to replace our current XML entries files with some
> variant of this new format. Below is three lines from an entries file
> to show how it might look:
>
> |dir|2|file:///home/pl/svntest/gcc/repo|file:///home/pl/svntest/gcc/repo||||2006-04-06T20:31:40.024395Z|2|pl|t||svn:special svn:externals svn:needs-lock||||||||||||b4f160ba-c4e8-11da-bcd8-65faf9067726|
> config-ml.in|file|||||2006-04-05T21:35:10.000000Z|e09724e0a7725cc08129db9293a35df9|2006-04-05T21:20:33.744115Z|1|pl|
> libgomp|dir|
> ...
>
> Each entry is on a separated line and each field is terminated by a |
> character. Empty fields may be omitted provided all following fields
> are also empty. Absent fields are represented by empty fields (except
> for the entry name, where an empty field means the thisdir entry).
> Boolean values are represent by "t" (for true" or an empty field for
> false. IN the future, we can add new fields at the end of each
> record. There will be an escaping mechanism for control characters
> (think newlines in lock comments) and the | separator.
Something that came up in conversation with jerenkrantz today...
Why not switch from pipe delimited to newline delimited fields? One
newline would delimit fields in an entry, a line with just a newline
on it would indicate a new entry. I suppose we might need some way to
indicate an empty field, but that should be easy enough, or heck, you
could even go based on the known number of fields in an entry, using
the blank line between entries as a sanity check. It would result in
far more easily readable entries files, and the size would be
identical...
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Justin Erenkrantz writes:
> On 4/7/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > Since the new format reduced the CPU time by about 35%, I think the
> > performance gain is a combination of I/O and CPU, especially when much
> > of the entries files is ion the OS cache, which is the case when you
> > perform multiple operations in a row. But I don't think this should
> > be very hard to try, so I'll give it a shot to get some perspective.
>
> There might be some opportunities to make our XML parsing a bit faster still.
>
> A few things looking at read_entries:
>
> 1) APR_BUFFERED isn't turned on, so any small reads or writes will
> cause extra libc/syscall activity. Combined with...
Will we normally have small reads here? We read into a reasonably
large buffer, so I'd think turning on APR_BUFFERED would just be
another memory copy operation (or does it read directly into large
buffers? I havent checked.)
> 2) No need to call read_full in the do loop. However many bytes it
> reads at a time is fine by us. read_full would cause multiple I/O ops
> of a non-optimal chunk size - the better strategy is to just process
> what we got from one libc/syscall invocation instead of forcing the
> return to be CHUNK_SIZE (16k).
I understand this for pipes and sockets, but not for regular files. I
don't think we normally see short reads on regular files unless we are
at EOF. Or what do I miss?
> It *might* be worth it to see if doing a mmap of the entries file
> would help - that would make the parsing faster as we can avoid a
> buffer copy entirely. Solaris, FreeBSD, and a few other OSes have a
> madvise option (MADV_SEQUENTIAL) that works with mmap that tells it as
> we read the memory location we can discard the previously read bytes
> and we will fetching the next bytes. This isn't insignficiant in some
> other usage patterns like this.
>
> Your recent changes to handle_start_tag to use a scratch pool helps
> save memory, but we could/should avoid the strdups that come with
> svn_xml_make_att_hash. It might just be worth teaching
> svn_wc__atts_to_entry how to handle the atts array directly to avoid
> the scratch pool altogether.
Yeah, looking at a profile shows that moving attributes into hashes
and looking them up is significant.
Still, we're spending lots of time in libexpat, according to oprofile.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 4/7/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Since the new format reduced the CPU time by about 35%, I think the
> performance gain is a combination of I/O and CPU, especially when much
> of the entries files is ion the OS cache, which is the case when you
> perform multiple operations in a row. But I don't think this should
> be very hard to try, so I'll give it a shot to get some perspective.
There might be some opportunities to make our XML parsing a bit faster still.
A few things looking at read_entries:
1) APR_BUFFERED isn't turned on, so any small reads or writes will
cause extra libc/syscall activity. Combined with...
2) No need to call read_full in the do loop. However many bytes it
reads at a time is fine by us. read_full would cause multiple I/O ops
of a non-optimal chunk size - the better strategy is to just process
what we got from one libc/syscall invocation instead of forcing the
return to be CHUNK_SIZE (16k).
It *might* be worth it to see if doing a mmap of the entries file
would help - that would make the parsing faster as we can avoid a
buffer copy entirely. Solaris, FreeBSD, and a few other OSes have a
madvise option (MADV_SEQUENTIAL) that works with mmap that tells it as
we read the memory location we can discard the previously read bytes
and we will fetching the next bytes. This isn't insignficiant in some
other usage patterns like this.
Your recent changes to handle_start_tag to use a scratch pool helps
save memory, but we could/should avoid the strdups that come with
svn_xml_make_att_hash. It might just be worth teaching
svn_wc__atts_to_entry how to handle the atts array directly to avoid
the scratch pool altogether.
The joys of staring at XML parsing code in ra_serf for the last few
days... Hah. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Daniel Berlin writes:
>
> On Apr 7, 2006, at 4:13 PM, Peter N. Lundblad wrote:
>
> > Mattias Engdegård writes:
> >> - Put dir-props and dir-prop-base into entries as well.
> >
> > This is normally just one extra file per directory. It will mean that
> > we will have to read and rewrite the dirprops every time we want to
> > update the entries file. NOte that the working props file is gone in
> > 1.4 unless there actually *are* property modifications. The same
> > holds for files.
> >
> >> - Combine all prop-files into one single file: Any file with
> >> properties will
> >> take a full disk block for each prop-file (two per file), in
> >> addition to
> >> the inodes (which are not free either, contrary to common belief).
> >> And it gets even worse when running SELinux[*].
> >
> > Two files is no noe, so the situation has improved. The problem with
> > combining all props into one file is that changing one property will
> > mean that we rewrite all properties just to set one property, which
> > might be a problem if you use large properties.
>
> This is only true if you use the file format we have now.
> The reality is we should stop using formats like that anyway (be it
> through using sqlite, or even some smarter text format).
We've been through this before, don't we? :-)
I agree, but don't see how that would happen in the near future.
Best,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Daniel Berlin <db...@dberlin.org>.
On Apr 7, 2006, at 4:13 PM, Peter N. Lundblad wrote:
> Mattias Engdegård writes:
>> - Put dir-props and dir-prop-base into entries as well.
>
> This is normally just one extra file per directory. It will mean that
> we will have to read and rewrite the dirprops every time we want to
> update the entries file. NOte that the working props file is gone in
> 1.4 unless there actually *are* property modifications. The same
> holds for files.
>
>> - Combine all prop-files into one single file: Any file with
>> properties will
>> take a full disk block for each prop-file (two per file), in
>> addition to
>> the inodes (which are not free either, contrary to common belief).
>> And it gets even worse when running SELinux[*].
>
> Two files is no noe, so the situation has improved. The problem with
> combining all props into one file is that changing one property will
> mean that we rewrite all properties just to set one property, which
> might be a problem if you use large properties.
This is only true if you use the file format we have now.
The reality is we should stop using formats like that anyway (be it
through using sqlite, or even some smarter text format).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: WC metadata redesign (was: [PROPOSAL] Drop XML from .svn/entries)
Posted by Vincent Lefevre <vi...@vinc17.org>.
On 2006-04-16 20:06:05 +0000, Mattias Engdegård wrote:
> Vincent Lefevre <vi...@vinc17.org> writes:
> >But users have reported that hard links aren't reliable yet under
> >Mac OS X.
>
> Do you have any references?
http://opendarwin.org/pipermail/darwinports/2006-March/033243.html
http://opendarwin.org/pipermail/darwinports/2006-March/033245.html
http://opendarwin.org/pipermail/darwinports/2006-April/033511.html
--
Vincent Lefèvre <vi...@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: WC metadata redesign (was: [PROPOSAL] Drop XML from .svn/entries)
Posted by Mattias Engdegård <ma...@virtutech.se>.
Vincent Lefevre <vi...@vinc17.org> writes:
>But users have reported that hard links aren't reliable yet under
>Mac OS X.
Do you have any references?
>One doesn't always use an editor. I often modify files with "cmd >>file".
Then you wouldn't use a hard-link option. And if you did so anyway by
mistake, the error would be detected (and possibly corrected) automatically.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: WC metadata redesign (was: [PROPOSAL] Drop XML from .svn/entries)
Posted by Vincent Lefevre <vi...@vinc17.org>.
On 2006-04-08 11:02:18 +0000, Mattias Engdegård wrote:
> > > - If the user explicitly asks for it, hardlinked text-base files.
>
> >This isn't portable and it is dangerous. I'd much more like optional
> >text bases to be implemented.
>
> Don't discount it that easily, it's more portable than you think.
> Hard links are available in OS X and even NTFS nowadays.
But users have reported that hard links aren't reliable yet under
Mac OS X.
> Dangerous, not really - as I wrote, corrupted text-base would easily
> be detected, and besides it would never be the default but an active
> user choice (svn co --I-promise-to-break-links-on-editing). All
> standard editors do the right thing by default.
One doesn't always use an editor. I often modify files with "cmd >>file".
Also this would have to be disabled for files with keywords (and possibly
with a different EOL). If this is optional, this can be OK.
--
Vincent Lefèvre <vi...@vinc17.org> - Web: <http://www.vinc17.org/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.org/blog/>
Work: CR INRIA - computer arithmetic / SPACES project at LORIA
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
WC metadata redesign (was: [PROPOSAL] Drop XML from .svn/entries)
Posted by Mattias Engdegård <ma...@virtutech.se>.
"Peter N. Lundblad" <pe...@famlundblad.se> writes:
> > - Put dir-props and dir-prop-base into entries as well.
>This is normally just one extra file per directory. It will mean that
>we will have to read and rewrite the dirprops every time we want to
>update the entries file. NOte that the working props file is gone in
>1.4 unless there actually *are* property modifications. The same
>holds for files.
It is admittedly a minor issue, but since the dirprops are typically
very small this should be no extra overhead.
> > - Combine all prop-files into one single file: [...]
>Two files is no noe, so the situation has improved. The problem with
s/is no noe/are no more/ ?
>combining all props into one file is that changing one property will
>mean that we rewrite all properties just to set one property, which
>might be a problem if you use large properties.
No, properties are _never_ large in practice; Subversion just isn't
used that way. I'm not advocating any artificial restrictions - users
can stuff gigabytes in their properties for all I care - but
optimising for the not merely uncommon but completely _nonexisting_
case is just bad engineering.
This came up some while ago when discussing whether MacOS resource
forks should be stored in properties, and the consensus was that it
was a Bad Idea for several reasons, size being one.
On the other hand, _small_ properties are not uncommon at all, which
makes the practice of putting them each in their own file(s) all the
more wasteful - 100 versioned files will easily eat 800 K just in
properties, no matter how small.
> I've been thinking of
>a compromise where you store the base props in one file (with some
>lookup table so you don't have to read the whole file to get one
>property) and then store working props in individual files when there
>are property mods. That's not going to happen for 1.4, though.
This sounds sensible.
> > - Use file locking instead of lock files. [...]
>[...]You need to keep a lock file open for each directory
>which might well exceed the per-process limit for maximum number of
>ope files. If we switch to a scheme where you lock the whole working
>copy, this will not be a performance problem anymore:-)
Agreed, locking the whole WC is by far preferable.
I profiled svn status/update to find out why it was so slow on NFS, and
the lock files stood out as an obvious problem (and NFS traffic confirmed
this), and especially on Solaris whose NFS implementation is more
protocol dogmatic (less caching) than Linux's.
> > - Combine all the .svn directories to one single directory in the
> > root of the WC. [...]
>This makes it impossible to move parts of a working copy to another
>place. I don't say this is the most important feature in the world (I
>don't know how many people actually use it), but it is a rather
>disruptive change.
I wasn't even aware of the possibility, and frankly I don't think it's
important at all (but would happily be corrected).
> > - If the user explicitly asks for it, hardlinked text-base files.
>This isn't portable and it is dangerous. I'd much more like optional
>text bases to be implemented.
Don't discount it that easily, it's more portable than you think.
Hard links are available in OS X and even NTFS nowadays. Dangerous,
not really - as I wrote, corrupted text-base would easily be detected,
and besides it would never be the default but an active user choice
(svn co --I-promise-to-break-links-on-editing). All standard editors
do the right thing by default.
Optional text bases are nice but slower and won't work when disconnected.
> > - Remove README.txt and empty-file.
>Done in 1.4.
Yay!
Thanks for the insightful reply. I realise a more thorough WC metadata
reform is a bit further ahead.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Mattias Engdegård writes:
> - Put dir-props and dir-prop-base into entries as well.
This is normally just one extra file per directory. It will mean that
we will have to read and rewrite the dirprops every time we want to
update the entries file. NOte that the working props file is gone in
1.4 unless there actually *are* property modifications. The same
holds for files.
> - Combine all prop-files into one single file: Any file with properties will
> take a full disk block for each prop-file (two per file), in addition to
> the inodes (which are not free either, contrary to common belief).
> And it gets even worse when running SELinux[*].
Two files is no noe, so the situation has improved. The problem with
combining all props into one file is that changing one property will
mean that we rewrite all properties just to set one property, which
might be a problem if you use large properties. I've been thinking of
a compromise where you store the base props in one file (with some
lookup table so you don't have to read the whole file to get one
property) and then store working props in individual files when there
are property mods. That's not going to happen for 1.4, though.
> - Use file locking instead of lock files. This should be considerably faster,
> especially over network file systems, and will not leave locks hanging if
> a crash occurs. (It won't work on NFSv2, but nor do plain lock files without
> a little bizarre dance with link() and stat() which isn't done today.)
Problematic with the current scheme where you can lock individual
directories. You need to keep a lock file open for each directory
which might well exceed the per-process limit for maximum number of
ope files. If we switch to a scheme where you lock the whole working
copy, this will not be a performance problem anymore:-)
> - Combine all the .svn directories to one single directory in the root of the
> WC. This could avoid several recursive traversals entirely, and need fewer
> locks. Users would be happy too (no .svn/ dirs everywhere).
This makes it impossible to move parts of a working copy to another
place. I don't say this is the most important feature in the world (I
don't know how many people actually use it), but it is a rather
disruptive change.
> - If the user explicitly asks for it, hardlinked text-base files.
> Most editors will break the links on editing, and disk usage would be
> almost halved. Inadvertent corruption of the text-base could be detected.
> Checkout performance would be much improved as it's mostly I/O bound.
> Comparing a file against its hard-link costs zero syscalls if the inodes
> (from the readdir) are the same.
This isn't portable and it is dangerous. I'd much more like optional
text bases to be implemented.
> - Remove README.txt and empty-file.
>
Done in 1.4.
Some of the ideas I dispute above are good, but not feasible
currently. If/when we rewrite the working copy library (or part of
it), I think we should consider a completely different way of storing
working copy metadataa. What I am trying to do now, is to improve the
current design while keeping its current feature set.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Mattias Engdegård <ma...@virtutech.se>.
Philip Martin <ph...@codematters.co.uk> writes:
>Reduced disk IO and a small reduction in working copy size. At
>present the format file is read every time the entries file is read.
Very worthwhile goals. If it is permitted to suggest more aggressive
WC improvements in the same spirit, I would like to mention:
- Put dir-props and dir-prop-base into entries as well.
- Combine all prop-files into one single file: Any file with properties will
take a full disk block for each prop-file (two per file), in addition to
the inodes (which are not free either, contrary to common belief).
And it gets even worse when running SELinux[*].
- Use file locking instead of lock files. This should be considerably faster,
especially over network file systems, and will not leave locks hanging if
a crash occurs. (It won't work on NFSv2, but nor do plain lock files without
a little bizarre dance with link() and stat() which isn't done today.)
- Combine all the .svn directories to one single directory in the root of the
WC. This could avoid several recursive traversals entirely, and need fewer
locks. Users would be happy too (no .svn/ dirs everywhere).
- If the user explicitly asks for it, hardlinked text-base files.
Most editors will break the links on editing, and disk usage would be
almost halved. Inadvertent corruption of the text-base could be detected.
Checkout performance would be much improved as it's mostly I/O bound.
Comparing a file against its hard-link costs zero syscalls if the inodes
(from the readdir) are the same.
- Remove README.txt and empty-file.
[*] SELinux, even in "permissive" mode, will tag each file with an
extended attribute. In ext3, extended attributes are stored in an
extra disk block. We discovered that a single medium-sized WC
consumed over 220 MB extra disk space this way - each file would
use 4K for itself, its text-base, prop and prop-base (even when
the latter two were empty).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch
Posted by Branko Čibej <br...@xbc.nu>.
Peter Samuelson wrote:
> [Marc Sherman]
>
>> What if, instead of moving the format number to the top of
>> .svn/entries, you got rid of the entries file and moved the
>> pipe-separated entries content to the bottom of the .svn/format file?
>>
>
> I'd hard-link the two filenames and fall back on writing a one-line
> 'format' file. Hard links, where available, are just a tiny bit more
> efficient than symlinks. That would require
>
> -AC_CHECK_FUNCS(symlink readlink)
> +AC_CHECK_FUNCS(symlink readlink link)
>
> in configure.in, and perhaps something a little fancier on win32 since
> hard links are only available on NTFS and possibly the API to create
> one is a little more involved than link(src, dest).
>
How often do hard links survive a "cp -R"?
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch
Posted by Peter Samuelson <pe...@p12n.org>.
[Marc Sherman]
> What if, instead of moving the format number to the top of
> .svn/entries, you got rid of the entries file and moved the
> pipe-separated entries content to the bottom of the .svn/format file?
I'd hard-link the two filenames and fall back on writing a one-line
'format' file. Hard links, where available, are just a tiny bit more
efficient than symlinks. That would require
-AC_CHECK_FUNCS(symlink readlink)
+AC_CHECK_FUNCS(symlink readlink link)
in configure.in, and perhaps something a little fancier on win32 since
hard links are only available on NTFS and possibly the API to create
one is a little more involved than link(src, dest).
Re: The nonxml-entries branch
Posted by Branko Čibej <br...@xbc.nu>.
Peter N. Lundblad wrote:
> Malcolm Rowe writes:
> > On Tue, Apr 11, 2006 at 05:17:40PM -0400, Marc Sherman wrote:
> > > Peter N. Lundblad wrote:
> > > > Why keep the .svn/format file?
> > > >
> > > > A disappointing thing with nonxml-entries is that we keep .svn/format,
> > > What if, instead of moving the format number to the top of .svn/entries,
> > > you got rid of the entries file and moved the pipe-separated entries
> > > content to the bottom of the .svn/format file?
> > >
> >
> > +1. That sounds like a really sensible idea, unless I'm missing something.
> >
>
> Technically, I see no problems implementing this. It just seems a little funny
> to store entries in a file called format. But if people feel comfortable
> with this, it is OK with me.
>
Please *no*.
It's a lot saner to move the format number to the beginning of the
entries file.
The parser can look at the first few (e.g., 5) bytes in the entries file
to discriminate between different format versions:
* It it sees "<?xml", then it knows that's a working copy as we have
it now, and that it must look at the format file.
* If it sees, e.g., "# ver", then it's a new working copy without
the format file, and the version number determines if the contents
are XML or not.
So as a first step, I'd propose moving the format number to .svn/entries
and ignoring (but not removing!) .svn/format. The header in the entries
file might look like this:
# version: 33
<?xml charset="utf-8">
and .svn/format would contain the same format number. We could do this
for 1.4.
Then, at the next minor release, we could remove the format file, bump
the format number, and for all I care stop using XML in the .svn/entries
file.
But let's *not* rush WC format changes.
Or, to put this a bit more in prespective: if we do decide (vote) to
rush head-over-heels into changing the entries format and removing the
format file, those who vote in favour should be required to subscribe to
the users@ list and handle all the screams.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Malcolm Rowe writes:
> On Tue, Apr 11, 2006 at 05:17:40PM -0400, Marc Sherman wrote:
> > Peter N. Lundblad wrote:
> > > Why keep the .svn/format file?
> > >
> > > A disappointing thing with nonxml-entries is that we keep .svn/format,
> > What if, instead of moving the format number to the top of .svn/entries,
> > you got rid of the entries file and moved the pipe-separated entries
> > content to the bottom of the .svn/format file?
> >
>
> +1. That sounds like a really sensible idea, unless I'm missing something.
>
Technically, I see no problems implementing this. It just seems a little funny
to store entries in a file called format. But if people feel comfortable
with this, it is OK with me.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch
Posted by Malcolm Rowe <ma...@farside.org.uk>.
On Tue, Apr 11, 2006 at 05:17:40PM -0400, Marc Sherman wrote:
> Peter N. Lundblad wrote:
> > Why keep the .svn/format file?
> >
> > A disappointing thing with nonxml-entries is that we keep .svn/format,
> What if, instead of moving the format number to the top of .svn/entries,
> you got rid of the entries file and moved the pipe-separated entries
> content to the bottom of the .svn/format file?
>
+1. That sounds like a really sensible idea, unless I'm missing something.
Regards,
Malcolm
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch
Posted by Marc Sherman <ms...@projectile.ca>.
Peter N. Lundblad wrote:
>
> Why keep the .svn/format file?
>
> A disappointing thing with nonxml-entries is that we keep .svn/format,
> at least for some minor svn releases. The reason for this is to let
> old clients give the user reasonably meaningful error messages when
> faced with a new working copy. It is better to say "your client is
> too old" than "This is not a working copy". We need to prefer
> usability to a small amount of disk space here.
What if, instead of moving the format number to the top of .svn/entries,
you got rid of the entries file and moved the pipe-separated entries
content to the bottom of the .svn/format file?
- Marc
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Philip Martin writes:
> "Peter N. Lundblad" <pe...@famlundblad.se> writes:
>
> [performace figures that look good]
>
> > I strongly feel this branch should be merged and would like some
> > review if possible.
>
> I've had a look at the code (not compiled or tested) and it looks good
> to me, I think it should be merged. Two comments so far:
Thanks for reviewing. As you may have noticed, I had to fix a case
where running cleanup on a WC in the old format would break the WC by
installing the new entries file in the new format, tithout upgrading
the rest of the WC. Do you care to take a look at the upgrading
strategy and see if it looks sane now?
> - there is broken indentation around line 1090 in entries.c.
Wow! Some obfuscation might not hurt, right? :-) F(Fixed now).
> - when writing the entries perhaps make a guess at the overall size
> (use the number of entries?) and pre-allocate bigstr to avoid some
> of the my_realloc calls.
>
I profiled a checkout (which is pretty write_entries intensive) and
I'm not sure it would make any difference. memcpy has about 7% of the
execution time, and that includes txdelta appplication and
libsvn_ra_svn protocol handling.. It might not hurt either, so I
wouldn't object...
Best Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch
Posted by Philip Martin <ph...@codematters.co.uk>.
"Peter N. Lundblad" <pe...@famlundblad.se> writes:
[performace figures that look good]
> I strongly feel this branch should be merged and would like some
> review if possible.
I've had a look at the code (not compiled or tested) and it looks good
to me, I think it should be merged. Two comments so far:
- there is broken indentation around line 1090 in entries.c.
- when writing the entries perhaps make a guess at the overall size
(use the number of entries?) and pre-allocate bigstr to avoid some
of the my_realloc calls.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch (was: Re: [PROPOSAL] Drop XML from .svn/entries)
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 4/18/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> Justin Erenkrantz writes:
> > On 4/11/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > > So, any review comments or whatever are as always welcome. Unless
> > > someone objects, I plan to merge this in a few days.
> >
> > If we merge this change now (which I have no real strong opinion
> > either way at this point), then I think it's prudent to push off
> > branching 1.4 for at a little bit longer - instead of at the end of
> > next week as I believe is currently planned. This can give us more
> > time to ensure we're as fast and as stable as we can be with this new
> > WC format instead of rushing it out the door. -- justin
> >
>
> OK, I'll try to get this branch merged tomorrow unless something shows up.
> Then I will commit the wcprops change (store all wcprops in one file
> per directory) soon after that.
>
> I propose we push back the branching date for 1.4 2 weeks to may
> 5th. Garrett, will you have some auth thingies ready for replay by then?
Hopefully, yes. At least enough to allow blocking of replay. I'm
pretty busy at work on non-svn stuff this week, but with luck I'll get
to it.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch (was: Re: [PROPOSAL] Drop XML from .svn/entries)
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Justin Erenkrantz writes:
> On 4/11/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> > So, any review comments or whatever are as always welcome. Unless
> > someone objects, I plan to merge this in a few days.
>
> If we merge this change now (which I have no real strong opinion
> either way at this point), then I think it's prudent to push off
> branching 1.4 for at a little bit longer - instead of at the end of
> next week as I believe is currently planned. This can give us more
> time to ensure we're as fast and as stable as we can be with this new
> WC format instead of rushing it out the door. -- justin
>
OK, I'll try to get this branch merged tomorrow unless something shows up.
Then I will commit the wcprops change (store all wcprops in one file
per directory) soon after that.
I propose we push back the branching date for 1.4 2 weeks to may
5th. Garrett, will you have some auth thingies ready for replay by then?
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: The nonxml-entries branch (was: Re: [PROPOSAL] Drop XML from .svn/entries)
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 4/11/06, Peter N. Lundblad <pe...@famlundblad.se> wrote:
> So, any review comments or whatever are as always welcome. Unless
> someone objects, I plan to merge this in a few days.
If we merge this change now (which I have no real strong opinion
either way at this point), then I think it's prudent to push off
branching 1.4 for at a little bit longer - instead of at the end of
next week as I believe is currently planned. This can give us more
time to ensure we're as fast and as stable as we can be with this new
WC format instead of rushing it out the door. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
The nonxml-entries branch (was: Re: [PROPOSAL] Drop XML from .svn/entries)
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Philip Martin writes:
> "Peter N. Lundblad" <pe...@famlundblad.se> writes:
>
> > Philip Martin writes:
> > > entries file. This would also seem to be an ideal opportunity to
> > > combine the format file with the entries file.
> >
> > Yeah, that sounds reasonable. What are the real benefits here?
>
> Reduced disk IO and a small reduction in working copy size. At
> present the format file is read every time the entries file is read.
> With your change more (most?) entries files will fit in a single disk
> block so getting rid of format file could halve the disk reads and
> reduce the pressure of the filesystem cache. Getting rid of the
> format file will reduce working copy disk usage by one or two percent
> in some cases.
>
OK, so I've been busy the last days implementing this functionality,
including combining .svn/format with .svn/entries. This is in
/branches/nonxml-entries for anyone interested.
I claim that a sequence of operations:
svn st
svn up (update with no significant changes to the tree)
sync (to avoid the previous operation interfering with the next)
svn st
svn diff
svn ci (with no modifications)
Are 25 - 30 % faster on the GCC tree now than before. The inclusion
of the format number in entries made the first of the operations above
significantly faster.
(According to a simple test, comparing nonxml-entries with an 1.3.1
client, we're now 60% faster on the sequence of commands listed above,
and this would be even more if the GCC made larger use of properties.)
There are further optimizations that could be done (for example,
currently, we open/read/close the first part of the entries file to
get the format number and then open/read/close the whole entries file
to get the entries). I'm focusing on the format change to be ready if
we include it in 1.4.
I strongly feel this branch should be merged and would like some
review if possible. I'll discuss some of the design choices I made
below.
Why not compress entries and keep it in XML?
I haven't tested this (yes I know I promised to), so I don't know how
much it would give us performance-wise. A problem with this aproach,
however, is that when we combine the format number with entries, it
would make us have to decompress the file to get to the working copy
format. This means we can't easily change the entries format with
regard to what/if to use compression in the future. With the version
number in plaintext on the first line, we can change the rest of the
file's format completely if we need to in the future.
Another suggestion was to use shorter attribute/element names in the
XML file, and perhaps trim some whitespace. If we want to include the
format number in the file, we face a similar problem to the one
described above. We can't just put the format number on the first
line, because then it wouldn't be a well-formed XML file anymore. We
could put the format number in a comment or processing instruction in
the prolog (before the first element), but then we would need to
either create an XML parser just to extract the version number, or do
some custom parsing, which can get complex.
Because of the above, and because I don't see any real benefit of
using XML in this particular case, I don't think any of the above
proposals are worth it.
Why keep the .svn/format file?
A disappointing thing with nonxml-entries is that we keep .svn/format,
at least for some minor svn releases. The reason for this is to let
old clients give the user reasonably meaningful error messages when
faced with a new working copy. It is better to say "your client is
too old" than "This is not a working copy". We need to prefer
usability to a small amount of disk space here.
So, any review comments or whatever are as always welcome. Unless
someone objects, I plan to merge this in a few days.
Regards,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Philip Martin <ph...@codematters.co.uk>.
"Peter N. Lundblad" <pe...@famlundblad.se> writes:
> Philip Martin writes:
> > entries file. This would also seem to be an ideal opportunity to
> > combine the format file with the entries file.
>
> Yeah, that sounds reasonable. What are the real benefits here?
Reduced disk IO and a small reduction in working copy size. At
present the format file is read every time the entries file is read.
With your change more (most?) entries files will fit in a single disk
block so getting rid of format file could halve the disk reads and
reduce the pressure of the filesystem cache. Getting rid of the
format file will reduce working copy disk usage by one or two percent
in some cases.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Philip Martin writes:
> "Peter N. Lundblad" <pe...@famlundblad.se> writes:
>
> > If we want this, we should do it before 1.4, since we already have WC
> > format upgrades.
>
> I think that is a bogus argument.
OK. Let me rephrase it like this:
I think we should either:
a) Get it into 1.4 since we already have WC formats bumps
or
b) Get it into the next minor release where we have a format bump for
some other reason.
I don't think this change warrants a format bump in itself, since the
costs to the users overweigh the benefits.
> I like it, I've always had doubts about the suitability of XML for the
> entries file. This would also seem to be an ideal opportunity to
> combine the format file with the entries file.
>
Yeah, that sounds reasonable. What are the real benefits here?
Julian Foad:
> Well, I have no particular objection to changing the format if it turns out to
> be a good idea after due consideration, but let me put the other side of the
> argument. Don't be too hasty. A 19% reduction in wall-clock time is worth
> considering, but not amazing.
>
Completely agree.
> Is this format that you are playing with one that you made up?
Well, you could call this a comma-separated database export format (but
the comma is a pipe) and then I don't claim to be the inventor:-)
But, yes, the specific details I have made up.
> Introducing Yet Another File Format is a bad thing in general,
> however simple it seems to be. There is significant but
> unquantifiable value in using a standard format. If a new format
> were to replace one of the other formats that we currently use, it
> would be more reasonable, but this won't be replacing XML.
Well, for interchange and communications, I agree about the general
statement above. But for an internal data format, I don't really see
the value of a standard format, especially since we don't use most of
its features. (I'm glad the days when XML should be used for
everything are over...)
We use custom formats in many places in our code already:
- FSFS has its own formats.
- The reporter in libsvn_repos as its own ad-hoc format for the spool
file.
- We have our famous hashdump format.
- The libsvn_ra_svn protocol.
- (And of course the svndiff format)
I don't claim that this validates yet another format, but OTOH, I
don't see why the entries file would benefit more from a standard
format than any of the other formats we have (maybe except for the
reporter spool file format which doesn't have any compatibility concerns).
>
> If, as you say, the savings might be mostly due to the reduced file size, then
> try compressing the files on the fly (reading/writing theough a zlib stream).
> Does that give a similar saving in wall-clock time? Presumably the CPU time
> would be increased, but is that important?
>
Since the new format reduced the CPU time by about 35%, I think the
performance gain is a combination of I/O and CPU, especially when much
of the entries files is ion the OS cache, which is the case when you
perform multiple operations in a row. But I don't think this should
be very hard to try, so I'll give it a shot to get some perspective.
> Basically, I think this is something to be considered but not
> something to be rushed in for v1.4 or that, on the present evidence,
> warrants a new file format. By all means post a patch and let it be
> tried and profiled a bit more.
I'll post a patch so we can compare numbers. As I said, before, maybe
I'm doing something wrong that favors my proposal:-) Other
OS/hardware characteristics might also be interesting.
Justin Erenkrantz:
> I wonder how the ConfigParser / INI format would do? We already use
> that in a bunch of other places and have all the parsing code in
> libsvn_subr. -- justin
That's basically a name=value format which isn't much different from
our flat XML format, so I don't think that'd give us anything. And I
really think the XML parser is more optimized than our config parser
(which is fine for our current use=).
Garrett Rooney:
> I wonder how much of the gain we could get just by making the xml less
> verbose. I've seen big speed gains in previous projects by taking
> verbose xml and making the tag/attribute names smaller (i.e. on the
> order of a single character each), it's less readable than the current
> xml format, but not as incomprehensible to the naked eye as the pipe
> separated one, and if it's really just size that's giving the speed
> gain it's worth looking at.
This should also be pretty simple and an interesting data point to have.
OK, it seems like I'll have some more profiling to do:-(
Thanks for the input, everyone,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Philip Martin <ph...@codematters.co.uk>.
"Peter N. Lundblad" <pe...@famlundblad.se> writes:
> If we want this, we should do it before 1.4, since we already have WC
> format upgrades.
I think that is a bogus argument.
> Opinions?
I like it, I've always had doubts about the suitability of XML for the
entries file. This would also seem to be an ideal opportunity to
combine the format file with the entries file.
--
Philip Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Branko Čibej <br...@xbc.nu>.
Justin Erenkrantz wrote:
> On 4/6/06, Garrett Rooney <ro...@electricjellyfish.net> wrote:
>
>> I think YAML is very cool, but I am totally -1 to adding a dependency
>> on a YAML parsing library to Subversion. We already need XML for
>> webdav anyway, so if we're going to use a 'standardized' format, we
>> might as well just use that.
>>
>
> I wonder how the ConfigParser / INI format would do? We already use
> that in a bunch of other places and have all the parsing code in
> libsvn_subr.
It woudl do just great, except that we'd probably have to optimize it a
bit (it currently does character-by-character reads from a stream, but a
buffered stream would fix that).
The really nice part is that, unlike the XML parser which must be taught
in advance about new tags when a format changes, the config file parser
doesn't have to know about section and value names. So we could embed
the entries version number in the file itself and interpret it /after/
we've parsed the file.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 4/6/06, Garrett Rooney <ro...@electricjellyfish.net> wrote:
> I think YAML is very cool, but I am totally -1 to adding a dependency
> on a YAML parsing library to Subversion. We already need XML for
> webdav anyway, so if we're going to use a 'standardized' format, we
> might as well just use that.
I wonder how the ConfigParser / INI format would do? We already use
that in a bunch of other places and have all the parsing code in
libsvn_subr. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 4/6/06, Blair Zajac <bl...@orcaware.com> wrote:
> Why not look at YAML?
>
> "YAML(tm) (rhymes with "camel") is a straightforward machine parsable data
> serialization format designed for human readability and interaction with
> scripting languages such as Perl and Python. YAML is optimized for data
> serialization, configuration settings, log files, Internet messaging and
> filtering. YAML(tm) is a balance of the following design goals.
>
> # YAML documents are very readable by humans.
> # YAML interacts well with scripting languages.
> # YAML uses host languages' native data structures.
> # YAML has a consistent information model.
> # YAML enables stream-based processing.
> # YAML is expressive and extensible.
> # YAML is easy to implement.
>
> It's already got implementations in most languages (Python, Ruby, Java, Perl)
> and I think it's got less overhead then XML.
>
> It would be interesting to see speed tests.
I think YAML is very cool, but I am totally -1 to adding a dependency
on a YAML parsing library to Subversion. We already need XML for
webdav anyway, so if we're going to use a 'standardized' format, we
might as well just use that.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Blair Zajac <bl...@orcaware.com>.
Peter N. Lundblad wrote:
> Hi,
>
> "Just For Fun", I experimented with replacing our current XML-based
> .svn/entries syntax with a line-based, |-separated syntax. It
> actually turns out that there is some overhead associated with using
> XML, but it is not extreme overhead.
>
> To get some idea how much overhead there actually is, I tested
> something that might be considered as a "typical work cycle". I used
> the following one-liner:
...
> If we want this, we should do it before 1.4, since we already have WC
> format upgrades.
>
> Opinions?
Why not look at YAML?
"YAML(tm) (rhymes with "camel") is a straightforward machine parsable data
serialization format designed for human readability and interaction with
scripting languages such as Perl and Python. YAML is optimized for data
serialization, configuration settings, log files, Internet messaging and
filtering. YAML(tm) is a balance of the following design goals.
# YAML documents are very readable by humans.
# YAML interacts well with scripting languages.
# YAML uses host languages' native data structures.
# YAML has a consistent information model.
# YAML enables stream-based processing.
# YAML is expressive and extensible.
# YAML is easy to implement.
It's already got implementations in most languages (Python, Ruby, Java, Perl)
and I think it's got less overhead then XML.
It would be interesting to see speed tests.
Regards,
Blair
--
Blair Zajac, Ph.D.
<bl...@orcaware.com>
Subversion training, consulting and support
http://www.orcaware.com/svn/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by John Peacock <jp...@rowman.com>.
Greg Hudson wrote:
> On Fri, 2006-04-07 at 00:53 +0100, Julian Foad wrote:
>> Is this format that you are playing with one that you made up? Introducing Yet
>> Another File Format is a bad thing in general, however simple it seems to be.
>> There is significant but unquantifiable value in using a standard format.
>
> I think this argument is overblown. A file format which appears to be
> simple often really is, and the code necessary to parse it is often less
> than the code necessary to interface to an XML library.
I have to agree. Although it isn't that common, one of my first programming
jobs involved a package (warehouse/order entry system) that spewed
pipe-delimited as its default output. The one big advantage was that the pipe
character was not used in the data itself, so there was never any escaping to
worry about. Here, we don't have that luxury, but it is much simpler to split
on a single character (modulo escapes) than any other file format. Human
readability can easily be produced with a quick one-liner in the scripting
language of your choice.
John
--
John Peacock
Director of Information Research and Technology
Rowman & Littlefield Publishing Group
4720 Boston Way
Lanham, MD 20706
301-459-3366 x.5010
fax 301-429-5747
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Greg Hudson <gh...@MIT.EDU>.
On Fri, 2006-04-07 at 00:53 +0100, Julian Foad wrote:
> Is this format that you are playing with one that you made up? Introducing Yet
> Another File Format is a bad thing in general, however simple it seems to be.
> There is significant but unquantifiable value in using a standard format.
I think this argument is overblown. A file format which appears to be
simple often really is, and the code necessary to parse it is often less
than the code necessary to interface to an XML library. Conversely, the
value in using a standard format almost never materializes in practice.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Steinar Bang <sb...@dod.no>.
>>>>> "Garrett Rooney" <ro...@electricjellyfish.net>:
> ZLib is probably worth trying, although I'd be somewhat against a
> compressed format due to readability issues...
FWIW, on my Ubuntu and debian systems, less and emacs (which are the
programs I use to look at files like these) unzip files on opening.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Garrett Rooney <ro...@electricjellyfish.net>.
On 4/6/06, Julian Foad <ju...@btopenworld.com> wrote:
> Well, I have no particular objection to changing the format if it turns out to
> be a good idea after due consideration, but let me put the other side of the
> argument. Don't be too hasty. A 19% reduction in wall-clock time is worth
> considering, but not amazing.
I don't know, I think it's a pretty nice speed bump for big trees...
> Is this format that you are playing with one that you made up? Introducing Yet
> Another File Format is a bad thing in general, however simple it seems to be.
> There is significant but unquantifiable value in using a standard format. If a
> new format were to replace one of the other formats that we currently use, it
> would be more reasonable, but this won't be replacing XML.
>
> If, as you say, the savings might be mostly due to the reduced file size, then
> try compressing the files on the fly (reading/writing theough a zlib stream).
> Does that give a similar saving in wall-clock time? Presumably the CPU time
> would be increased, but is that important?
ZLib is probably worth trying, although I'd be somewhat against a
compressed format due to readability issues...
I wonder how much of the gain we could get just by making the xml less
verbose. I've seen big speed gains in previous projects by taking
verbose xml and making the tag/attribute names smaller (i.e. on the
order of a single character each), it's less readable than the current
xml format, but not as incomprehensible to the naked eye as the pipe
separated one, and if it's really just size that's giving the speed
gain it's worth looking at.
-garrett
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Julian Foad <ju...@btopenworld.com>.
Peter N. Lundblad wrote:
> "Just For Fun", I experimented with replacing our current XML-based
> .svn/entries syntax with a line-based, |-separated syntax. It
> actually turns out that there is some overhead associated with using
> XML, but it is not extreme overhead.
[...]
> The result is that this row of operation requires 19% less wallclock
> time and 35% less CPU time. [...]
>
> ON another note, the size of all the entries files in the GCC tree
> decreased from 8.2 to 4.2 megabytes, which probably explains much of
> the time savings.
[...]
> So, I propose to replace our current XML entries files with some
> variant of this new format.
[...]
> Opinions?
Well, I have no particular objection to changing the format if it turns out to
be a good idea after due consideration, but let me put the other side of the
argument. Don't be too hasty. A 19% reduction in wall-clock time is worth
considering, but not amazing.
Is this format that you are playing with one that you made up? Introducing Yet
Another File Format is a bad thing in general, however simple it seems to be.
There is significant but unquantifiable value in using a standard format. If a
new format were to replace one of the other formats that we currently use, it
would be more reasonable, but this won't be replacing XML.
If, as you say, the savings might be mostly due to the reduced file size, then
try compressing the files on the fly (reading/writing theough a zlib stream).
Does that give a similar saving in wall-clock time? Presumably the CPU time
would be increased, but is that important?
Basically, I think this is something to be considered but not something to be
rushed in for v1.4 or that, on the present evidence, warrants a new file
format. By all means post a patch and let it be tried and profiled a bit more
by other people, and also work out the escaping details and post the formal
description of the format, and we'll take it from there. Documenting a design
is usually a very good way of showing up its strengths and weaknesses.
- Julian
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Michael Brouwer <mb...@gmail.com>.
Last time I checked, bdb doesn't work on non local filesystems. And
subversion definitely needs to support checkout to NFS, SMB, AFP and
other types of remote filesystems that people might use as home dirs.
sqlite3 might be an option though and has already been suggested here.
Michael
On 4/22/06, Arlie Davis <ar...@insightbb.com> wrote:
> I think this entire proposal to move away from XML and to an ad-hoc format
> is a very bad one.
>
> I've seen many projects use XML for large datasets, and do so very
> efficiently. Why not try to improve the existing implementation, rather
> than inventing a new format and going through all the unnecessary pain?
>
> The key to XML (or anything) is usually understanding the data flow of the
> most important performance-sensitive operations, and then structuring your
> implementation around it. If you are concerned about entries files that are
> large enough that using an XML DOM is painful, then use one of the streaming
> XML implementations, which give you a great deal of control over how the
> data is read/written, and when.
>
> Abandoning XML without truly compelling reasons is a very bad idea. And if
> performance is so critical, then replace the XML entries file with a
> Berkeley DB file (btree), rather than building yet another hacked-up text
> format. At least with BDB, there's already a solid implementation.
>
> -- arlie
>
>
>
>
> -----Original Message-----
> From: Peter N. Lundblad [mailto:peter@famlundblad.se]
> Sent: Friday, April 21, 2006 2:03 AM
> To: Peter Samuelson
> Cc: dev@subversion.tigris.org
> Subject: Re: [PROPOSAL] Drop XML from .svn/entries
>
> Peter Samuelson writes:
> >
> > [Jonathan Gilbert]
> > > Why not add *support* for the new format to the core libraries right >
> > away, but not actually create/upgrade repositories at this > > point.
> >
> > Because maintaining the ability to read *and* write multiple repository
> > formats at once is annoying. The normal mode is to read the old format >
> but only write the new format.
>
> In general, yes, but it depends on what the format change is. The recent
> changes (nonxml-entries and storing wcprops in a single file) require the
> odl write support to stay in the code so a new client can run cleanup on an
> old WC (we don't try to upgrade a WC that needs cleanup). For
> wcpropcaching, the situation is different because the code that runs the
> logs doesn't deal with the way properties are stored directly. So, it gets a
> little tricky... And everybody who knows how stupid I really am now fears
> 1.4:-)
>
> Best,
> //Peter
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
RE: [PROPOSAL] Drop XML from .svn/entries
Posted by Arlie Davis <ar...@insightbb.com>.
I think this entire proposal to move away from XML and to an ad-hoc format
is a very bad one.
I've seen many projects use XML for large datasets, and do so very
efficiently. Why not try to improve the existing implementation, rather
than inventing a new format and going through all the unnecessary pain?
The key to XML (or anything) is usually understanding the data flow of the
most important performance-sensitive operations, and then structuring your
implementation around it. If you are concerned about entries files that are
large enough that using an XML DOM is painful, then use one of the streaming
XML implementations, which give you a great deal of control over how the
data is read/written, and when.
Abandoning XML without truly compelling reasons is a very bad idea. And if
performance is so critical, then replace the XML entries file with a
Berkeley DB file (btree), rather than building yet another hacked-up text
format. At least with BDB, there's already a solid implementation.
-- arlie
-----Original Message-----
From: Peter N. Lundblad [mailto:peter@famlundblad.se]
Sent: Friday, April 21, 2006 2:03 AM
To: Peter Samuelson
Cc: dev@subversion.tigris.org
Subject: Re: [PROPOSAL] Drop XML from .svn/entries
Peter Samuelson writes:
>
> [Jonathan Gilbert]
> > Why not add *support* for the new format to the core libraries right >
> away, but not actually create/upgrade repositories at this > > point.
>
> Because maintaining the ability to read *and* write multiple repository
> formats at once is annoying. The normal mode is to read the old format >
but only write the new format.
In general, yes, but it depends on what the format change is. The recent
changes (nonxml-entries and storing wcprops in a single file) require the
odl write support to stay in the code so a new client can run cleanup on an
old WC (we don't try to upgrade a WC that needs cleanup). For
wcpropcaching, the situation is different because the code that runs the
logs doesn't deal with the way properties are stored directly. So, it gets a
little tricky... And everybody who knows how stupid I really am now fears
1.4:-)
Best,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Peter Samuelson writes:
>
> [Jonathan Gilbert]
> > Why not add *support* for the new format to the core libraries right
> > away, but not actually create/upgrade repositories at this
> > point.
>
> Because maintaining the ability to read *and* write multiple repository
> formats at once is annoying. The normal mode is to read the old format
> but only write the new format.
In general, yes, but it depends on what the format change is. The
recent changes (nonxml-entries and storing wcprops in a single file)
require the odl write support to stay in the code so a new client can
run cleanup on an old WC (we don't try to upgrade a WC that needs
cleanup). For wcpropcaching, the situation is different because the
code that runs the logs doesn't deal with the way properties are
stored directly. So, it gets a little tricky... And everybody who
knows how stupid I really am now fears 1.4:-)
Best,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Peter Samuelson <pe...@p12n.org>.
[Jonathan Gilbert]
> Why not add *support* for the new format to the core libraries right
> away, but not actually create/upgrade repositories at this
> point.
Because maintaining the ability to read *and* write multiple repository
formats at once is annoying. The normal mode is to read the old format
but only write the new format.
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Jonathan Gilbert <o2...@sneakemail.com>.
At 08:33 PM 14/04/2006 +0200, Peter N. Lundblad wrote:
>Justin Erenkrantz writes:
> > On 4/14/06, Branko ?ibej <br...@xbc.nu> wrote:
> > > /me wonders if there is some deep, universal truth evident to everybody
> > > but himself
> >
> > I think Peter means that we shouldn't bump the WC format between
> > public minor revs if we don't have to. Altering entries to a non-XML
> > format *would* bump WC's format - but if we did it before 1.4, as far
> > as everyone who uses our packaged releases is concerned, we just had a
> > format bump between 1.3 and 1.4. (Only dev's would have seen the
> > current interim changes; their WC's would be upgraded too - but for
> > people using public releases, they'd see the format in 1.3 and then
> > have their format bumped to the non-XML format w/1.4 - bypassing the
> > format we're at right now...) -- justin
>
>Couldn't say it better myself:-) When we put out 1.4 prereleases, we
>will have to include warnings that if you test this release, you can't
>easily go back to use 1.3 on this working copy. This, and also the
>fact that users who start using 1.4 when it is released might not be
>able to mix that with other clients that haven't upgraded to the 1.4
>libraries yet, is very annoying. I don't see a way around it, because
>completely supporting old formats is a maintainanc nightmare, but we
>should avoid bumping if possible.
Why not add *support* for the new format to the core libraries right away,
but not actually create/upgrade repositories at this point. Then, when 1.4
comes out, presumably other clients will have had the opportunity to add
the necessary support (for many, simply recompiling against the updated
libraries, no?).
Jonathan Gilbert
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
Justin Erenkrantz writes:
> On 4/14/06, Branko ?ibej <br...@xbc.nu> wrote:
> > /me wonders if there is some deep, universal truth evident to everybody
> > but himself
>
> I think Peter means that we shouldn't bump the WC format between
> public minor revs if we don't have to. Altering entries to a non-XML
> format *would* bump WC's format - but if we did it before 1.4, as far
> as everyone who uses our packaged releases is concerned, we just had a
> format bump between 1.3 and 1.4. (Only dev's would have seen the
> current interim changes; their WC's would be upgraded too - but for
> people using public releases, they'd see the format in 1.3 and then
> have their format bumped to the non-XML format w/1.4 - bypassing the
> format we're at right now...) -- justin
Couldn't say it better myself:-) When we put out 1.4 prereleases, we
will have to include warnings that if you test this release, you can't
easily go back to use 1.3 on this working copy. This, and also the
fact that users who start using 1.4 when it is released might not be
able to mix that with other clients that haven't upgraded to the 1.4
libraries yet, is very annoying. I don't see a way around it, because
completely supporting old formats is a maintainanc nightmare, but we
should avoid bumping if possible.
Thanks,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On 4/14/06, Branko Čibej <br...@xbc.nu> wrote:
> Wait, do you suggest to change the entries file format an /not/ change
> the WC format number?
>
> /me wonders if there is some deep, universal truth evident to everybody
> but himself
I think Peter means that we shouldn't bump the WC format between
public minor revs if we don't have to. Altering entries to a non-XML
format *would* bump WC's format - but if we did it before 1.4, as far
as everyone who uses our packaged releases is concerned, we just had a
format bump between 1.3 and 1.4. (Only dev's would have seen the
current interim changes; their WC's would be upgraded too - but for
people using public releases, they'd see the format in 1.3 and then
have their format bumped to the non-XML format w/1.4 - bypassing the
format we're at right now...) -- justin
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Branko Čibej <br...@xbc.nu>.
Peter N. Lundblad wrote:
> In another mail, you suggest that we should move the format number
> inot entries now and then bump it in 1.5 to replace the format of the
> file. I am strongly -1 on this strategy, unless we have some other
> good reason to bump the format number. Bumping the format is not nice
> to peoople who for some reason have to use mixed client versions, so
> lets not do that unless really necessary.
>
Wait, do you suggest to change the entries file format an /not/ change
the WC format number?
/me wonders if there is some deep, universal truth evident to everybody
but himself
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by "Peter N. Lundblad" <pe...@famlundblad.se>.
brane@xbc.nu writes:
> Before we go ballistic about file formats, I'd like to note that the
> real problem with XML in .svn/entries isn't speed but all the escaping
> that needs to be done. Escaping will be needed with any format, but XML
> is a special PITA here.
If you claim that using XML does not affect performance, then just do
a profile and see for yourself.. The verbosity of the format leads to
much more IO than necessary and CPU usage as well.
> I'd also like to point out that we have a config file parser that's
> quite good enough for entries files, so please don't invent another file
> format.
That format has some pecularities that make it unsuitable for this
purpose. One example is the locale-dependent stripping of whitespace
around values. Also, why go from one name=value format to another, as
I pointed out in another mail.
> That parser may need some optimisations if we use it for entries
> files, but at least it's not completely new code.
>
And it currently parses the file into many small hash tables, which is
quite wasteful. The XML parsing code does the same, which shows up in
the profiles.
In another mail, you suggest that we should move the format number
inot entries now and then bump it in 1.5 to replace the format of the
file. I am strongly -1 on this strategy, unless we have some other
good reason to bump the format number. Bumping the format is not nice
to peoople who for some reason have to use mixed client versions, so
lets not do that unless really necessary.
Best,
//Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Daniel Berlin <db...@dberlin.org>.
On Apr 12, 2006, at 6:00 AM, Branko Čibej wrote:
> Before we go ballistic about file formats, I'd like to note that
> the real problem with XML in .svn/entries isn't speed but all the
> escaping that needs to be done. Escaping will be needed with any
> format, but XML is a special PITA here.
Actually, speed is a problem.
We currently have to rewrite the entire file to change a single entry
right now.
You could probably hack around that by just appending if you could
guarantee that cutting the existing xml and appending xml entries
would never cause problems.
Certainly, for svn 2.0, we shouldn't be using file formats that don't
support replace and update without rewriting the entire thing.
(How we do this, i don't care).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: [PROPOSAL] Drop XML from .svn/entries
Posted by Branko Čibej <br...@xbc.nu>.
Before we go ballistic about file formats, I'd like to note that the
real problem with XML in .svn/entries isn't speed but all the escaping
that needs to be done. Escaping will be needed with any format, but XML
is a special PITA here.
I'd also like to point out that we have a config file parser that's
quite good enough for entries files, so please don't invent another file
format. That parser may need some optimisations if we use it for entries
files, but at least it's not completely new code.
-- Brane
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org