You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by "Hyrum K. Wright" <hy...@mail.utexas.edu> on 2009/04/15 21:29:42 UTC
Making wc-ng the default on trunk
Currently, trunk creates and uses format 11 working copies, which are
"old style". These have the standard entries file, along with various
directories for wcprops, regular props, pristine files, and other
bits. In order to make implementation easier, and to help us test wc-
ng, I'd like to move trunk to format 12, the wc-ng format, soon.
Here's why, and what you need to know for the transition.
Pros:
* Eating our own dog food is a Good Thing.
* We can remove the current cruft in trunk which is used for
upgrading and writing to format 11 working copies. (Reading code will
still hang around because it is required as part of the upgrade
process.)
* Generally more straight forward implementation, as we focus
strictly on the new format.
Cons:
* Possible breakage as bugs are worked out (though, if you run
trunk, you assume that's part of the deal, right? :) )
* The format implementation is still in flux, which may cause
incompatibilities within format 12 versions (see breakage, above).
* Current implementation is *slow*.
Stuff that needs to happen before we can make the switch:
* Remaining test failures with wc-ng on trunk fixes (6 left).
* Implement some cache cleanup, as the number of open database
handles currently exceeds the number allowed open file descriptors on
some systems.
* Show we can do a checkout and commit using trunk.
I think Greg is going to look at the caching issues at some point, and
I pray he looks at the test failures too. We've implemented an
upgrade solution which will allow conversion of existing working
copies to the new format. We also plan to warn people again when we
make the switch, and possibly give folks a few days to test stuff "in
the wild" before we cut over.
Questions?
-Hyrum
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1733271
Re: Making wc-ng the default on trunk
Posted by Mark Phippard <ma...@gmail.com>.
On Thu, Apr 16, 2009 at 9:29 AM, Hyrum K. Wright
<hy...@mail.utexas.edu> wrote:
> We can go ahead and try this, but forgive me if I don't have much
> faith that it will happen. We had serf on trunk for *years* and
> encouraged people to use it, but it wasn't until it because the
> default on trunk that folks really started testing it. We can tell
> people to use the envvar for wc-ng, but I don't think it will happen
> on a regular basis.
I thought Stefan was just echoing what you were suggesting. When you
think WC is "ready" send a message telling people how they can try it,
and wait a week or so before making it the default. Basically, just
expand the real world testing before we make it the default. Isn't
that what you meant when you said:
"We also plan to warn people again when we
make the switch, and possibly give folks a few days to test stuff "in
the wild" before we cut over."
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1747622
Re: Making wc-ng the default on trunk
Posted by "Hyrum K. Wright" <hy...@mail.utexas.edu>.
On Apr 15, 2009, at 5:52 PM, Stefan Sperling wrote:
> On Wed, Apr 15, 2009 at 04:29:42PM -0500, Hyrum K. Wright wrote:
>> Currently, trunk creates and uses format 11 working copies, which are
>> "old style". These have the standard entries file, along with
>> various
>> directories for wcprops, regular props, pristine files, and other
>> bits. In order to make implementation easier, and to help us test
>> wc-
>> ng, I'd like to move trunk to format 12, the wc-ng format, soon.
>> Here's why, and what you need to know for the transition.
>
> +1 here, but can we first make everyone of us run with SVN_ENABLE_NG
> for a week or two, before we make it the default? I'd hate to run into
> show-stopper bugs that make working copies unusable without having a
> convenient way to revert to wc-1.
>
> Since it's so easy to run wc-ng by just setting an env var, I don't
> think anyone would not try it if you said "everyone, please set the
> env var now, we think it's ready". And that might already uncover
> some bugs.
We can go ahead and try this, but forgive me if I don't have much
faith that it will happen. We had serf on trunk for *years* and
encouraged people to use it, but it wasn't until it because the
default on trunk that folks really started testing it. We can tell
people to use the envvar for wc-ng, but I don't think it will happen
on a regular basis.
-Hyrum
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1747567
Re: Making wc-ng the default on trunk
Posted by Stefan Sperling <st...@elego.de>.
On Wed, Apr 15, 2009 at 04:29:42PM -0500, Hyrum K. Wright wrote:
> Currently, trunk creates and uses format 11 working copies, which are
> "old style". These have the standard entries file, along with various
> directories for wcprops, regular props, pristine files, and other
> bits. In order to make implementation easier, and to help us test wc-
> ng, I'd like to move trunk to format 12, the wc-ng format, soon.
> Here's why, and what you need to know for the transition.
+1 here, but can we first make everyone of us run with SVN_ENABLE_NG
for a week or two, before we make it the default? I'd hate to run into
show-stopper bugs that make working copies unusable without having a
convenient way to revert to wc-1.
Since it's so easy to run wc-ng by just setting an env var, I don't
think anyone would not try it if you said "everyone, please set the
env var now, we think it's ready". And that might already uncover
some bugs.
Stefan
RE: Making wc-ng the default on trunk
Posted by Bert Huijben <rh...@sharpsvn.net>.
> -----Original Message-----
> From: Hyrum K. Wright [mailto:hyrum_wright@mail.utexas.edu]
> Sent: woensdag 15 april 2009 23:30
> To: dev@subversion.tigris.org
> Subject: Making wc-ng the default on trunk
>
> Currently, trunk creates and uses format 11 working copies, which are
> "old style". These have the standard entries file, along with various
> directories for wcprops, regular props, pristine files, and other
> bits. In order to make implementation easier, and to help us test wc-
> ng, I'd like to move trunk to format 12, the wc-ng format, soon.
> Here's why, and what you need to know for the transition.
>
> Pros:
> * Eating our own dog food is a Good Thing.
> * We can remove the current cruft in trunk which is used for
> upgrading and writing to format 11 working copies. (Reading code will
> still hang around because it is required as part of the upgrade
> process.)
> * Generally more straight forward implementation, as we focus
> strictly on the new format.
>
> Cons:
> * Possible breakage as bugs are worked out (though, if you run
> trunk, you assume that's part of the deal, right? :) )
On Windows that would be certain breakage as the failed test count is still
above 100 (120 last Monday; 118 just now) As far as I can see mostly for the
single reason that Subversion keeps handles to .svn/wc.db open when the file
is deleted for removing the directory.
All subversion tests/actions that remove one or more working copy
directories just FAIL and there are a lot of tests and user scenarios that
add or remove directories.
This should be fixed before (or /really/ short after) switching to WC-NG as
the default working copy format. One solution would be to move everything to
the single working copy database, but I don't think you planned to do that
around this switch?
> * The format implementation is still in flux, which may cause
> incompatibilities within format 12 versions (see breakage, above).
> * Current implementation is *slow*.
>
> Stuff that needs to happen before we can make the switch:
> * Remaining test failures with wc-ng on trunk fixes (6 left).
On Windows that number is much higher...
I would like to switch to WC-NG, but in the current state that is not an
option for me or any other Windows user :(
Summary of test results:
956 tests PASSED
43 tests SKIPPED
29 tests XFAILED (1 WORK-IN-PROGRESS)
118 tests FAILED
3 tests XPASSED (3 WORK-IN-PROGRESS)
If somebody needs more details:
http://b.qqn.nl/f/2009/wcng-20090615-summary.txt
http://b.qqn.nl/f/2009/wcng-20090615.txt (9MB)
http://b.qqn.nl/f/2009/wcng-20090615.txt.gz (450 KByte)
Bert
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1735654
Re: Making wc-ng the default on trunk
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Apr 16, 2009 at 15:35, Mark Phippard <ma...@gmail.com> wrote:
> On Thu, Apr 16, 2009 at 9:32 AM, Hyrum K. Wright
> <hy...@mail.utexas.edu> wrote:
>> On Apr 16, 2009, at 4:33 AM, Branko Cibej wrote:
>>
>>> Hyrum K. Wright wrote:
>>>> * Current implementation is *slow*.
>>>>
>>>
>>> I assume you know why and how to fix that, right?
>>
>> Aside from the sqlite database issues mentioned by Greg elsethread,
>> we're also not taking full advantage of the new data store yet.
>> Neither properties or the dav cache (the old 'wcprops') have moved
>> into the sqlite database. Almost all the code external to wc-ng is
>> still relying upon svn_wc_entry_t to interface with the sqlite
>> database, which requires a complex, and potentially performance
>> intensive, translation. Moving the rest of libsvn_wc to hit the wc_db
>> directly will make things quicker indeed.
>
> So you are mainly wanting to do this to avoid the costly work of
> supporting the interim format 11. Aren't you going to have the same
> problem with 12 and others? As you move more things to the database
> and eventually get rid of the per-directory .svn folders aren't you
> going to continue to have these problems to deal with?
The bulk of the data is *already* in the database. It's just the
properties that are external. IOW, there are no "more things" other
than properties.
Right now, we can't really use the wc_db without some costly work.
I've enabled a couple APIs (read_info and read_children), but the rest
are off limits. For each one, we have to write code to map that
functionality onto the v11 format. Ugh. After that, then we can start
using the new API within libsvn_wc. In short: it is very slow going to
*actually* move libsvn_wc onto v12 since we still have to support v11.
Once v11 is dropped, then we can call wc_db APIs at will (those that
are implemented, at least).
Cheers,
-g
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1747926
Re: Making wc-ng the default on trunk
Posted by Mark Phippard <ma...@gmail.com>.
On Thu, Apr 16, 2009 at 9:32 AM, Hyrum K. Wright
<hy...@mail.utexas.edu> wrote:
> On Apr 16, 2009, at 4:33 AM, Branko Cibej wrote:
>
>> Hyrum K. Wright wrote:
>>> * Current implementation is *slow*.
>>>
>>
>> I assume you know why and how to fix that, right?
>
> Aside from the sqlite database issues mentioned by Greg elsethread,
> we're also not taking full advantage of the new data store yet.
> Neither properties or the dav cache (the old 'wcprops') have moved
> into the sqlite database. Almost all the code external to wc-ng is
> still relying upon svn_wc_entry_t to interface with the sqlite
> database, which requires a complex, and potentially performance
> intensive, translation. Moving the rest of libsvn_wc to hit the wc_db
> directly will make things quicker indeed.
So you are mainly wanting to do this to avoid the costly work of
supporting the interim format 11. Aren't you going to have the same
problem with 12 and others? As you move more things to the database
and eventually get rid of the per-directory .svn folders aren't you
going to continue to have these problems to deal with?
--
Thanks
Mark Phippard
http://markphip.blogspot.com/
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1747658
Re: Making wc-ng the default on trunk
Posted by "Hyrum K. Wright" <hy...@mail.utexas.edu>.
On Apr 16, 2009, at 4:33 AM, Branko Cibej wrote:
> Hyrum K. Wright wrote:
>> * Current implementation is *slow*.
>>
>
> I assume you know why and how to fix that, right?
Aside from the sqlite database issues mentioned by Greg elsethread,
we're also not taking full advantage of the new data store yet.
Neither properties or the dav cache (the old 'wcprops') have moved
into the sqlite database. Almost all the code external to wc-ng is
still relying upon svn_wc_entry_t to interface with the sqlite
database, which requires a complex, and potentially performance
intensive, translation. Moving the rest of libsvn_wc to hit the wc_db
directly will make things quicker indeed.
-Hyrum
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1747602
Re: Making wc-ng the default on trunk
Posted by Stefan Sperling <st...@elego.de>.
On Thu, Apr 16, 2009 at 11:33:29AM +0200, Branko Cibej wrote:
> Hyrum K. Wright wrote:
> > * Current implementation is *slow*.
> >
>
> I assume you know why and how to fix that, right?
Yes!
Buy a faster computer.
Stefan
Re: Making wc-ng the default on trunk
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Apr 16, 2009 at 14:00, Arfrever Frehtes Taifersar Arahesis
<ar...@gmail.com> wrote:
> 2009-04-16 13:53 Greg Stein <gs...@gmail.com> napisał(a):
>> On Thu, Apr 16, 2009 at 11:33, Branko Cibej <br...@xbc.nu> wrote:
>>> Hyrum K. Wright wrote:
>>>> * Current implementation is *slow*.
>>>
>>> I assume you know why and how to fix that, right?
>>
>> Because we still open a SQLite database per directory. LOTS of times
>> for the same file. When we get rid of the extraneous open/close
>> behavior, and (especially) migrate to a single DB, then it will be
>> much faster.
>
> How much work is still required to migrate to a single database?
A ton.
I believe that we have to eliminate almost all uses of
svn_wc_adm_access_t from our codebase before we can do the migration.
We'll be reducing the number of open/close cycles as a natural result
of the elimination. So performance should steadily improve during that
work.
Cheers,
-g
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1746663
Re: Making wc-ng the default on trunk
Posted by Arfrever Frehtes Taifersar Arahesis <Ar...@GMail.Com>.
2009-04-16 13:53 Greg Stein <gs...@gmail.com> napisał(a):
> On Thu, Apr 16, 2009 at 11:33, Branko Cibej <br...@xbc.nu> wrote:
>> Hyrum K. Wright wrote:
>>> * Current implementation is *slow*.
>>
>> I assume you know why and how to fix that, right?
>
> Because we still open a SQLite database per directory. LOTS of times
> for the same file. When we get rid of the extraneous open/close
> behavior, and (especially) migrate to a single DB, then it will be
> much faster.
How much work is still required to migrate to a single database?
--
Arfrever Frehtes Taifersar Arahesis
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1746117
Re: Making wc-ng the default on trunk
Posted by Greg Stein <gs...@gmail.com>.
On Thu, Apr 16, 2009 at 11:33, Branko Cibej <br...@xbc.nu> wrote:
> Hyrum K. Wright wrote:
>> * Current implementation is *slow*.
>
> I assume you know why and how to fix that, right?
Because we still open a SQLite database per directory. LOTS of times
for the same file. When we get rid of the extraneous open/close
behavior, and (especially) migrate to a single DB, then it will be
much faster.
That said, we haven't started on any performance analysis whatsoever.
So in a sense, "no, we don't know how to fix it."
Cheers,
-g
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1745929
Re: Making wc-ng the default on trunk
Posted by Branko Cibej <br...@xbc.nu>.
Hyrum K. Wright wrote:
> * Current implementation is *slow*.
>
I assume you know why and how to fix that, right?
------------------------------------------------------
http://subversion.tigris.org/ds/viewMessage.do?dsForumId=462&dsMessageId=1743930