You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Daniel Patterson <da...@adaptiveinternational.com> on 2003/06/23 04:58:33 UTC
OT: Configuration management for stateful data
Greetings all,
I'm looking for advice on the best practises for configuration
management of stateful data.
This situation comes up in a few different places, but the
best example (and the one I have right now) are database
schema changes between software releases.
For stateless portions of the system (i.e. the application
code), it's easy to overwrite, or drop/replace (in the case
of database stored procedures). However, stateful portions
(i.e. database tables with data in them, configuration files,
any other stateful information that may undergo a schema change),
the usual configuration management procedures don't apply.
You can't drop 12 months of data from your database because
you need to rename a relational table....
A bit of searching around on google didn't reveal anything,
but it's hard to know if I'm searching for the wrong thing or
if nothing really exists.
I've thought a bit about the process that Subversion takes
(a dump into a known format followed by a reload), but in
our situation, building a tool like that is impractical.
Has anyone done any deep thinking about this topic and come
up with any good patterns to follow?
daniel
--
Signature at: http://www.mel.au.adaptiveinternational.com/~danpat/signature.txt
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: OT: Configuration management for stateful data
Posted by Tim Hawkins <ti...@timhawkins.co.uk>.
I tend to use database dumps (POSTGRESQL and MYSQL) in the repository for
test data, but not for production data. Its good to be able to recover a
test database that is synced to a particular code revision.
However the test dumps I use tend to be 1 or 2 megs at the most and are all
ascii.
----- Original Message -----
From: "Bruce Elrick" <br...@entropyreduction.ca>
To: "Bruce Elrick" <br...@entropyreduction.ca>
Cc: "Daniel Patterson" <da...@adaptiveinternational.com>;
<de...@subversion.tigris.org>
Sent: Monday, June 23, 2003 2:51 PM
Subject: Re: OT: Configuration management for stateful data
> Holy smokes my typing is bad...sorry for all the typos. I should go
> back to bed.
>
> Bruce Elrick wrote:
>
> > I know R/3 does table copies within the database (as opposed to dump &
> > load). That is, copy current table to temporary, drop original,
> > rebuild table with new properties, copy from temporary to new, drop
> > temporary. Obviously the last portion must application/business
> > contain logic.
>
> must contain application/business logic.
>
> > If the table changes are simple (add a field that is allowed to be
> > null) and the RDBMS allows it, you make be able to simply change the
> > table in situ.
>
> may be able to
>
> > However, even such a simple change may be procluded if having a large
> > number of the old records with nulls in the new field makes the
> > application less useful.
>
> precluded
>
> > For example, you might actually have to run a data cleansing project
> > where you get humans to go through that data and provide values for
> > the new field on a record-by-record basis. Maybe you can do that
> > before the code upgrade, maybe you can do it after.
> > In terms of whether it is practical, I don't think it really matters:
> > it is necessary. If anything stops you from doing it, then you can't
> > change upgrade/change, full stop.
> >
> > Cheers...
> > Bruce
> >
> > Daniel Patterson wrote:
> >
> >> Greetings all,
> >>
> >> I'm looking for advice on the best practises for configuration
> >> management of stateful data.
> >>
> >> This situation comes up in a few different places, but the
> >> best example (and the one I have right now) are database
> >> schema changes between software releases.
> >>
> >> For stateless portions of the system (i.e. the application
> >> code), it's easy to overwrite, or drop/replace (in the case
> >> of database stored procedures). However, stateful portions
> >> (i.e. database tables with data in them, configuration files,
> >> any other stateful information that may undergo a schema change),
> >> the usual configuration management procedures don't apply.
> >> You can't drop 12 months of data from your database because
> >> you need to rename a relational table....
> >>
> >> A bit of searching around on google didn't reveal anything,
> >> but it's hard to know if I'm searching for the wrong thing or
> >> if nothing really exists.
> >>
> >> I've thought a bit about the process that Subversion takes
> >> (a dump into a known format followed by a reload), but in
> >> our situation, building a tool like that is impractical.
> >>
> >> Has anyone done any deep thinking about this topic and come
> >> up with any good patterns to follow?
> >>
> >> daniel
> >>
> >>
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> > For additional commands, e-mail: dev-help@subversion.tigris.org
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: OT: Configuration management for stateful data
Posted by Bruce Elrick <br...@entropyreduction.ca>.
Holy smokes my typing is bad...sorry for all the typos. I should go
back to bed.
Bruce Elrick wrote:
> I know R/3 does table copies within the database (as opposed to dump &
> load). That is, copy current table to temporary, drop original,
> rebuild table with new properties, copy from temporary to new, drop
> temporary. Obviously the last portion must application/business
> contain logic.
must contain application/business logic.
> If the table changes are simple (add a field that is allowed to be
> null) and the RDBMS allows it, you make be able to simply change the
> table in situ.
may be able to
> However, even such a simple change may be procluded if having a large
> number of the old records with nulls in the new field makes the
> application less useful.
precluded
> For example, you might actually have to run a data cleansing project
> where you get humans to go through that data and provide values for
> the new field on a record-by-record basis. Maybe you can do that
> before the code upgrade, maybe you can do it after.
> In terms of whether it is practical, I don't think it really matters:
> it is necessary. If anything stops you from doing it, then you can't
> change upgrade/change, full stop.
>
> Cheers...
> Bruce
>
> Daniel Patterson wrote:
>
>> Greetings all,
>>
>> I'm looking for advice on the best practises for configuration
>> management of stateful data.
>>
>> This situation comes up in a few different places, but the
>> best example (and the one I have right now) are database
>> schema changes between software releases.
>>
>> For stateless portions of the system (i.e. the application
>> code), it's easy to overwrite, or drop/replace (in the case
>> of database stored procedures). However, stateful portions
>> (i.e. database tables with data in them, configuration files,
>> any other stateful information that may undergo a schema change),
>> the usual configuration management procedures don't apply.
>> You can't drop 12 months of data from your database because
>> you need to rename a relational table....
>>
>> A bit of searching around on google didn't reveal anything,
>> but it's hard to know if I'm searching for the wrong thing or
>> if nothing really exists.
>>
>> I've thought a bit about the process that Subversion takes
>> (a dump into a known format followed by a reload), but in
>> our situation, building a tool like that is impractical.
>>
>> Has anyone done any deep thinking about this topic and come
>> up with any good patterns to follow?
>>
>> daniel
>>
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: OT: Configuration management for stateful data
Posted by Bruce Elrick <br...@entropyreduction.ca>.
I know R/3 does table copies within the database (as opposed to dump &
load). That is, copy current table to temporary, drop original, rebuild
table with new properties, copy from temporary to new, drop temporary.
Obviously the last portion must application/business contain logic. If
the table changes are simple (add a field that is allowed to be null)
and the RDBMS allows it, you make be able to simply change the table in
situ. However, even such a simple change may be procluded if having a
large number of the old records with nulls in the new field makes the
application less useful. For example, you might actually have to run a
data cleansing project where you get humans to go through that data and
provide values for the new field on a record-by-record basis. Maybe you
can do that before the code upgrade, maybe you can do it after.
In terms of whether it is practical, I don't think it really matters: it
is necessary. If anything stops you from doing it, then you can't
change upgrade/change, full stop.
Cheers...
Bruce
Daniel Patterson wrote:
>Greetings all,
>
> I'm looking for advice on the best practises for configuration
> management of stateful data.
>
> This situation comes up in a few different places, but the
> best example (and the one I have right now) are database
> schema changes between software releases.
>
> For stateless portions of the system (i.e. the application
> code), it's easy to overwrite, or drop/replace (in the case
> of database stored procedures). However, stateful portions
> (i.e. database tables with data in them, configuration files,
> any other stateful information that may undergo a schema change),
> the usual configuration management procedures don't apply.
> You can't drop 12 months of data from your database because
> you need to rename a relational table....
>
> A bit of searching around on google didn't reveal anything,
> but it's hard to know if I'm searching for the wrong thing or
> if nothing really exists.
>
> I've thought a bit about the process that Subversion takes
> (a dump into a known format followed by a reload), but in
> our situation, building a tool like that is impractical.
>
> Has anyone done any deep thinking about this topic and come
> up with any good patterns to follow?
>
>daniel
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: %HOME% on Win32 (Re: ra_dav compression question)
Posted by Branko Čibej <br...@xbc.nu>.
Chris Hecker wrote:
> You can already change your %APPDATA% setting to, say, a subdirectory of
>
>> your %HOME%. So you can get almost exactly what you want without
>> changing a single line of SVN code.
>
>
> This will destroy the universe, I'm fairly sure. Windows is really
> bad about that kind of thing. I've never tried it, however. Have you?
Yes, I have. It works. And these days, Windows is really _good_ about
these things; only programs that don't conform to the Windows
conventions have problems, and that'w what I want to avoid.
>> However, Subversion also _creates_ that directory; how does it know
>> which you prefer?
>
>
> Sorry, I wasn't clear. I was saying that if it finds that directory
> is not there, it will check for %HOME%/.subversion as well. That's
> all. So, most of the time it will just use %APPDATA% for people who
> don't care, but if somebody does then they'll move the directory and
> svn will do the right thing.
I see. So you create %HOME%/.subversion before running SVN for the first
time, nad make sure the %APPDATA%/Subversion doesn't exist, and from
that point on things should just work?
Hm.
> From looking at it briefly, it'll be very little code to make this
> change, only a few lines in one function
> (svn_config__user_config_path) and one line in the config_impl.h
> header to do it right.
Oh, no. All of this logic should go into config_win.c, and nowhere else.
It's _different_ than on Unix, even with just the directory check.
> That won't work, .lnk files are handled by the Explorer, not the
>
>> filesystem. However, on Win2k and above, you can make
>> %APPDATA%\Subversion a junction to anywhere you like -- again, not
>> having to change a single line of code in SVN.
>
>
> Except junctions aren't very well documented or supported
Yes, they are, if you look at the right place. Also, the "dir" command
will show junctions.
> and there are reports of filesystem weirdness when using them.
What kind? I've been using them for years, with no percieved weirdness.
The main thing is that they're supported on the filesystem level. The
drawback is that they're NTFS-only, but if you ask me, people who don't
use NTFS are in for trouble anyway.
> Again, I'm happy to write the 10 lines of code and test them and send
> a patch to the HEAD, I just want to know it will be considered before
> I do that. If you're absolutely not interested I won't bother.
> However, I think the feature is sound and this is the best of the
> options.
This discussion is similar to the one about whether the SVN command-line
client should print paths with \ or / as a separator. As always, my
first answer is to conform to the OS native conventions.
What you're proposing complicates the code, but it's just possible that
the complication isn't too big. So, let's see the patch first, and
decide afterwards.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: %HOME% on Win32 (Re: ra_dav compression question)
Posted by Chris Hecker <ch...@d6.com>.
>Exactly. Unix ports. Subversion on Windows is _not_ a Unix port, believe
>it or not.
Sure, that's why I put the parenthetical comment in there. My point is
there's going to be a high correlation between people who have %HOME%
defined and people who use svn.exe. But anyway, I'm not saying to break
the old way, I'm saying to check for the new way in addition. I don't
think it will be much code. And, as a side benefit, the 8 zillion comments
in the code referring to ~/.subversion will suddenly be optionally
applicable to Win32 (since you're _not_ a unix port, this might appeal to
you :).
>You can already change your %APPDATA% setting to, say, a subdirectory of
>your %HOME%. So you can get almost exactly what you want without
>changing a single line of SVN code.
This will destroy the universe, I'm fairly sure. Windows is really bad
about that kind of thing. I've never tried it, however. Have you? But
even if it did work, it's not what you want; you don't want a huge slew of
random windows apps writing into your home directory, you want your
"unix-like" command line tools to do so. So, this should be a subversion
specific thing, I think.
>Note that
>%HOME% is not a standard env. var on Windows, so you're proposing a
>"new" variable in any case. :-)
>However, Subversion also _creates_ that directory; how does it know
>which you prefer?
Sorry, I wasn't clear. I was saying that if it finds that directory is not
there, it will check for %HOME%/.subversion as well. That's all. So, most
of the time it will just use %APPDATA% for people who don't care, but if
somebody does then they'll move the directory and svn will do the right thing.
From looking at it briefly, it'll be very little code to make this change,
only a few lines in one function (svn_config__user_config_path) and one
line in the config_impl.h header to do it right.
>That won't work, .lnk files are handled by the Explorer, not the
>filesystem. However, on Win2k and above, you can make
>%APPDATA%\Subversion a junction to anywhere you like -- again, not
>having to change a single line of code in SVN.
Except junctions aren't very well documented or supported and there are
reports of filesystem weirdness when using them. The .lnk suggestion was
because a number of windows apps are starting to parse .lnk as a symbolic
link transparently since junctions aren't first class citizens. It's not
hard, but it is more windows cruft. The directory check is way simpler and
cleaner, I think.
Again, I'm happy to write the 10 lines of code and test them and send a
patch to the HEAD, I just want to know it will be considered before I do
that. If you're absolutely not interested I won't bother. However, I
think the feature is sound and this is the best of the options.
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: %HOME% on Win32 (Re: ra_dav compression question)
Posted by Branko Čibej <br...@xbc.nu>.
Chris Hecker wrote:
>
>> > PPS. Special bonus question that I haven't had time to look in the
>> > source to answer: for the Win32 client, is there any way to use the
>> > %HOME%\.subversion directory rather than \Documents and Settings\blah
>> > blah blah\Subverion directory.
>> No. How many points do I get for this answer? :-)
>
>
> How many points you get depends on which "no" it is. Is that "no" as
> in "no, that's not currently possible but if somebody wrote the
> code...", or "no" as in "we will absolutely never consider such a
> feature"? :) If it's the former I'd be happy to submit a patch and
> you get full points.
Heh. Whatever gets me more points, then. :-)
> Putting per-user settings in %APPADATA%\Subversion is simply following
>
>> the Windows conventions, same as putting them in ~/.subversion is
>> following the Unix conventions. It's also necessary to do so if you want
>> the SVN settings to work correctly with roaming profiles and such.
>
>
> Sure, but there are a zillion other command line tools that use %HOME%
> (many many more than use the Windows conventions, like emacs, cvs,
> cygwin tools, openssl, etc...all unix ports, to be sure,
Exactly. Unix ports. Subversion on Windows is _not_ a Unix port, believe
it or not.
> but also highly correlated with commandline source code control tool
> use as well, I'd assume), so I claim giving people the option to use
> %HOME% if they choose is the right thing. Anybody who goes and toasts
> their %APPDATA%\Subversion directory and also has a %HOME%\.subversion
> directory is already making a choice to manage their user information
> themselves.
You can already change your %APPDATA% setting to, say, a subdirectory of
your %HOME%. So you can get almost exactly what you want without
changing a single line of SVN code.
> Or, another alternative is having a %SVNHOME% envvar, and the Win32
> default install can stuff it with %APPDATA% (like it does with the
> APR_ICONV_PATH)
APR_ICONV_PATH is not a Subversion thing, it's an apr-iconv thing, and
(unfortunately) necessary.
> and users who want to can change it. I think silently testing
> existence on %APPDATA%\Subversion and %HOME%\.subversion is better
> than yet another environment variable,
Absolutely, adding new environment variables is horrible. Note that
%HOME% is not a standard env. var on Windows, so you're proposing a
"new" variable in any case. :-)
However, Subversion also _creates_ that directory; how does it know
which you prefer?
> but either way is fine with me as long as it's possible to put the
> configs in %HOME%\.subversion. Or, how about following a link if
> %APPDATA%\Subversion is a .lnk. Lots of possibilities.
That won't work, .lnk files are handled by the Explorer, not the
filesystem. However, on Win2k and above, you can make
%APPDATA%\Subversion a junction to anywhere you like -- again, not
having to change a single line of code in SVN.
I think a bit of documentation on the things you can already do -- i.e.,
move %APPDATA% or make a junction -- would be the best solution here. We
really don't want to complicate the configuration code more than it is
already.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
%HOME% on Win32 (Re: ra_dav compression question)
Posted by Chris Hecker <ch...@d6.com>.
> > PPS. Special bonus question that I haven't had time to look in the
> > source to answer: for the Win32 client, is there any way to use the
> > %HOME%\.subversion directory rather than \Documents and Settings\blah
> > blah blah\Subverion directory.
>No. How many points do I get for this answer? :-)
How many points you get depends on which "no" it is. Is that "no" as in
"no, that's not currently possible but if somebody wrote the code...", or
"no" as in "we will absolutely never consider such a feature"? :) If it's
the former I'd be happy to submit a patch and you get full points.
>Putting per-user settings in %APPADATA%\Subversion is simply following
>the Windows conventions, same as putting them in ~/.subversion is
>following the Unix conventions. It's also necessary to do so if you want
>the SVN settings to work correctly with roaming profiles and such.
Sure, but there are a zillion other command line tools that use %HOME%
(many many more than use the Windows conventions, like emacs, cvs, cygwin
tools, openssl, etc...all unix ports, to be sure, but also highly
correlated with commandline source code control tool use as well, I'd
assume), so I claim giving people the option to use %HOME% if they choose
is the right thing. Anybody who goes and toasts their %APPDATA%\Subversion
directory and also has a %HOME%\.subversion directory is already making a
choice to manage their user information themselves. Or, another
alternative is having a %SVNHOME% envvar, and the Win32 default install can
stuff it with %APPDATA% (like it does with the APR_ICONV_PATH) and users
who want to can change it. I think silently testing existence on
%APPDATA%\Subversion and %HOME%\.subversion is better than yet another
environment variable, but either way is fine with me as long as it's
possible to put the configs in %HOME%\.subversion. Or, how about following
a link if %APPDATA%\Subversion is a .lnk. Lots of possibilities.
Thoughts?
Thanks,
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Branko Čibej <br...@xbc.nu>.
Chris Hecker wrote:
> PPS. Special bonus question that I haven't had time to look in the
> source to answer: for the Win32 client, is there any way to use the
> %HOME%\.subversion directory rather than \Documents and Settings\blah
> blah blah\Subverion directory.
No. How many points do I get for this answer? :-)
> Maybe if the latter doesn't exist but the former does or
> something...it's much more convenient to use %HOME% if you're already
> using it for a lot of other stuff.
Putting per-user settings in %APPADATA%\Subversion is simply following
the Windows conventions, same as putting them in ~/.subversion is
following the Unix conventions. It's also necessary to do so if you want
the SVN settings to work correctly with roaming profiles and such.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: ra_dav compression question
Posted by Chris Hecker <ch...@d6.com>.
>The one who added the input filtering was Justin. For full history, see:
> http://cvs.apache.org/viewcvs/httpd-2.0/modules/experimental/mod_deflate.c
>Specifically look at r1.10 where it was introduced.
Hah, that's ironic. Okay, so based on that, I'm assuming Justin meant "you
can't negotiate it, so you just do it when you know the server supports it"
and "you know" because we've set some bidirectional compression flag in the
svn config to tell neon to compress on output (with the code I'll
write). Or something like that.
Unfortunately, to do this right on the svn side means either having
"http-compression=yes" always mean bidi (which is bad because some people
might not SetInputFilter), or splitting it to "http-request-compression"
and "http-response-compression", or having more than "yes/no", so you'd
have "http-compression=yes|no|request|response".
Chris
PS. The 1.10 rev you're referring to is in the filters directory, not
experimental (as in your url). You actually checked in 1.10 in the
experimental directory, which confused me for a minute. :)
http://cvs.apache.org/viewcvs.cgi/httpd-2.0/modules/filters/mod_deflate.c
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
RE: ra_dav compression question
Posted by Sander Striker <st...@apache.org>.
> From: Chris Hecker [mailto:checker@d6.com]
> Sent: Tuesday, June 24, 2003 3:11 AM
> >>bidirectionally without a SetInputFilter. Anyway, I can probably do the
> >>work on neon, but I still don't know how to negotiate the compression. Any
> >>ideas on that?
> >As I've been trying to say, you can't. -- justin
>
> So it's in apache why, exactly? And how do the other WebDAV clients do it
> (perhaps I should try to contact that mod_deflate author and find out for
> sure)?
The one who added the input filtering was Justin. For full history, see:
http://cvs.apache.org/viewcvs/httpd-2.0/modules/experimental/mod_deflate.c
Specifically look at r1.10 where it was introduced.
Sander
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Chris Hecker <ch...@d6.com>.
>>bidirectionally without a SetInputFilter. Anyway, I can probably do the
>>work on neon, but I still don't know how to negotiate the compression. Any
>>ideas on that?
>As I've been trying to say, you can't. -- justin
So it's in apache why, exactly? And how do the other WebDAV clients do it
(perhaps I should try to contact that mod_deflate author and find out for
sure)? Or, are you just saying you can't negotiate it, so you just do it
and send Content-Encoding: gzip (based on some per-server svn config
setting mirroring http-compression) and hope it works, or optionally resend
if it doesn't (although that might be harder if the data's not around anymore)?
Chris
PS. I found the User-Agent problem in ssl.conf (it disables keepalive for
all MSIE browsers on SSL connections).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Monday, June 23, 2003 5:10 PM -0700 Chris Hecker <ch...@d6.com> wrote:
> bidirectionally without a SetInputFilter. Anyway, I can probably do the
> work on neon, but I still don't know how to negotiate the compression. Any
> ideas on that?
As I've been trying to say, you can't. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Chris Hecker <ch...@d6.com>.
>What SetInputFilter docs are you reading? I'm not aware of any current
>web browser supports sending compressed request bodies.
http://httpd.apache.org/docs-2.0/mod/mod_deflate.html#enable
It says no browsers support it but some WebDAV clients do. I couldn't find
any reference to one that did, but I didn't look for more than a few minutes.
>Plus, the server might only allow compressed request bodies based on the
>request-uri.
Right, but then I don't see how the compressed request is negotiated at all
from the bit I've looked into it. The link above implies you just send it
gzip and see what happens, but that's just my interpretation, it's not
clear from the docs.
>>Hmm, from looking into this more, it looks like any Content-Encoding: gzip
>>response from apache is Connection: close (even if you requested
>>keep-alive), which not only means you can't do the keepalive thing I
>If you can detail why you believe this to be true, please enlighten
>me. None of my servers exhibit this behavior. Are you sure you are using
>HTTP/1.1 connections? (HTTP/1.0 defaults to Connection: close)
Yeah, it was HTTP/1.1. Based on it not happening on your servers, I looked
at it more and if I remove the User-Agent from my test header (grabbed from
MSIE 6) it keeps the connection alive, so it must be a BrowserMatch thing,
even though I can't see one that possibly matches with a nokeepalive
directive in my default httpd.conf. I'll look at it more, but that's good
to know it was my mistake and not something inherent with keepalive
there. Still, since apache won't send the Accept-Encoding: gzip back with
a response we're back to stage one on how to negotiate the gzip request.
>>Also, the server doesn't seem to send back Accept-Encoding: gzip on
>>replies, so you probably just have to try a Content-Encoding: gzip
>>request and see if it doesn't error back to decide if you can do it.
>You'd be trying to handle negotiation outside of the protocol. That'd be
>extremely poor behavior.
Agreed, but I don't see any way to negotiate it that's documented in my
[admittedly brief and clueless] look into this.
>In practice, the majority of the data that an SVN client would send in a
>request body would be compressed via the vdelta algorithm or be extremely
>short XML request bodies.
Except during adds, or big file changes. I agree with you for projects
with a lot of people checking in and then you update to get their changes,
but if you check in a lot of changes/adds yourself then none of your
checkins are compressed. Imagine a single person using subversion for
revision control...in that use-case, none of the file data communication is
ever compressed (beyond the "compression" in vdelta), because that single
person is doing all the changing and adding and never getting any new data
from the server, so it's all client -> server unless he updates to an older
revision.
>I don't believe there would be an extreme performance win by compressing
>the request bodies. I could be proven wrong - someone would just have to
>add compressed request bodies to neon. As long as you are talking to a
>known server that has 'SetInputFilter DEFLATE' set, it should all work
>transparently. -- justin
Yeah, I don't know if it would be a huge win either (although you'd think
on adds it would make a bit difference), I just started looking into it as
a new user and couldn't figure out how it could possibly compress
bidirectionally without a SetInputFilter. Anyway, I can probably do the
work on neon, but I still don't know how to negotiate the compression. Any
ideas on that?
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Monday, June 23, 2003 3:12 PM -0700 Chris Hecker <ch...@d6.com> wrote:
> Sure, but how do the mysterious other DAV clients mentioned near the
> SetInputFilter docs handle this? I'd assume that on the first reply from
What SetInputFilter docs are you reading? I'm not aware of any current web
browser supports sending compressed request bodies.
> uncompressed data going to the server before the compression started. Does
> neon keep the connection alive and keep state about it so that this would be
> possible?
Having support for compressed request bodies and compressed response bodies
are separate. Making an assumption that the two are always linked would be a
very bad thing to do. Plus, the server might only allow compressed request
bodies based on the request-uri.
> Hmm, from looking into this more, it looks like any Content-Encoding: gzip
> response from apache is Connection: close (even if you requested
> keep-alive), which not only means you can't do the keepalive thing I
If you can detail why you believe this to be true, please enlighten me. None
of my servers exhibit this behavior. Are you sure you are using HTTP/1.1
connections? (HTTP/1.0 defaults to Connection: close)
> Also, the server doesn't seem to send back Accept-Encoding: gzip on replies,
> so you probably just have to try a Content-Encoding: gzip request and see if
> it doesn't error back to decide if you can do it.
You'd be trying to handle negotiation outside of the protocol. That'd be
extremely poor behavior.
> since it is not bidirectional like it's implied. And, the performance
> tradeoff of sending Accept-Encoding: gzip and dealing with the zillion
> reconnects versus not sending it and having a keepalive connection would be
> interesting to look into (assuming I'm right about that, and by someone with
> more http experience than me :).
In practice, the majority of the data that an SVN client would send in a
request body would be compressed via the vdelta algorithm or be extremely
short XML request bodies.
I don't believe there would be an extreme performance win by compressing the
request bodies. I could be proven wrong - someone would just have to add
compressed request bodies to neon. As long as you are talking to a known
server that has 'SetInputFilter DEFLATE' set, it should all work
transparently. -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Chris Hecker <ch...@d6.com>.
>Neon could be modified to send compressed request bodies. But, that's
>more difficult because there is no 'handshake' to ensure that the server
>can handle compressed request bodies. With responses, the client sends
>the 'Accept-Encoding: gzip' in the request, so there is a 'handshake' to
>ensure that both sides can support deflate. So, compressing request
>bodies isn't a realistic expectation to support with HTTP.
Sure, but how do the mysterious other DAV clients mentioned near the
SetInputFilter docs handle this? I'd assume that on the first reply from
the server it would send an Accept-Encoding: gzip back, and from that point
on neon would compress on that connection. This assumes the connections
are kept alive so that the second+ requests on the same connection can be
compressed. I assume svn always does a small request at the start anyway
to get a clue about the current situation, so there'd only be a tiny bit of
uncompressed data going to the server before the compression started. Does
neon keep the connection alive and keep state about it so that this would
be possible?
Hmm, from looking into this more, it looks like any Content-Encoding: gzip
response from apache is Connection: close (even if you requested
keep-alive), which not only means you can't do the keepalive thing I
mentioned above on a single connection, but also that _every_ request
answered with compressed data closes the connection on apache2 (I assume
because that's how apache tells you the compressed data is done since
Content-Length is the uncompressed size, even though if the receiver was
smart the uncompressed size would be enough info). For https this really
sucks, because it means the SSL connection is renegotiated on every
request, which has got to be really slow for updating a huge repository,
although I haven't timed it. Am I missing something here (hopefully), or
is this totally broken, performance-wise for https?
Also, the server doesn't seem to send back Accept-Encoding: gzip on
replies, so you probably just have to try a Content-Encoding: gzip request
and see if it doesn't error back to decide if you can do it.
It seems like this is fairly important to either fix or mention in the
manual near the place where "wire compression" is claimed when using
apache, since it is not bidirectional like it's implied. And, the
performance tradeoff of sending Accept-Encoding: gzip and dealing with the
zillion reconnects versus not sending it and having a keepalive connection
would be interesting to look into (assuming I'm right about that, and by
someone with more http experience than me :).
>Using ra_svn with ssh tunneling with compression enabled would do both
>ways as it blindly does compression on all data.
Right, but it seems like you'd want to keep the ra_dav path as feature-ful
as the ra_svn path, since it has a lot of other advantages as claimed in
the manual (including not having to give people shell accounts to give them
repository access, which is huge).
Thanks!
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Branko Čibej <br...@xbc.nu>.
Chris Hecker wrote:
>
>> Quite a lot, actually. The last numbers I got are that we're within 10%
>> of "diff -e | gzip -9", and we'll beat that consistently once Dan
>> Berlin's svndiff patch goes in (i.e., once I get the time and
>> inclination to finally bring it up to date).
>
>
> That's good. Is it run on adds?
I don't believe so; I'm not too familiar with that part of the code.
> How is that compression ratio possible unless you're actually doing a
> real compression (not just diff/delta)
We are doing real compression; it's part of the vdelta algorithm.
> on it in the first place, and if you are, then why would you want
> SetOutputFilter on there anyway (since you'd risk bloating from
> double-compression, maybe, or at the very least wasting cpu trying to
> compress it again)?
>
>> Yeah. And I'm wondering if compressing our delta stream is really worth
>> the effort.
>
>
> Indeed. Has anyone checked to see whether using mod_deflate is a lose
> from recompressing in the first place, if it's that well compressed
> already?
It would be interesting to measure that, yes. New data (from checkouts
and added files) would definitely benefit from mod_deflate, at least
most of the time. Then there are all sorts of XML reports that compress
really well.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Chris Hecker <ch...@d6.com>.
>Quite a lot, actually. The last numbers I got are that we're within 10%
>of "diff -e | gzip -9", and we'll beat that consistently once Dan
>Berlin's svndiff patch goes in (i.e., once I get the time and
>inclination to finally bring it up to date).
That's good. Is it run on adds? How is that compression ratio possible
unless you're actually doing a real compression (not just diff/delta) on it
in the first place, and if you are, then why would you want SetOutputFilter
on there anyway (since you'd risk bloating from double-compression, maybe,
or at the very least wasting cpu trying to compress it again)?
>Yeah. And I'm wondering if compressing our delta stream is really worth
>the effort.
Indeed. Has anyone checked to see whether using mod_deflate is a lose from
recompressing in the first place, if it's that well compressed already?
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Branko Čibej <br...@xbc.nu>.
cmpilato@collab.net wrote:
>Chris Hecker <ch...@d6.com> writes:
>
>
>
>>A delta/diff is different from a compressed stream. Sure, it's not
>>sending the parts that didn't change, but the changes aren't
>>compressed (and, as someone said, adds are the special case where it
>>all changes), if I understand things correctly. Not that sending
>>diffs isn't great, but you really want the compression to go both
>>ways, especially since that's part of the reason one is supposed to
>>use apache as the server from the manual ("wire compression for free").
>>
>>
>
>Generally speaking, you're correct. But Subversion uses a binary
>diffing format that not only sends just the differences, but even
>those differences are transmitted in a gzip-like fashion. I forget
>the stats someone figured out, but there definitely is some
>compression going on.
>
Quite a lot, actually. The last numbers I got are that we're within 10%
of "diff -e | gzip -9", and we'll beat that consistently once Dan
Berlin's svndiff patch goes in (i.e., once I get the time and
inclination to finally bring it up to date).
> In other words, if your differences are that
>you added the phrase "Chris is a pleasant programmer" in ten different
>places through a file, the trasmitted data might list that string only
>once, and then refer to it ten times.
>
It gets even better than that -- the two "is " in there would get
compressed to one, probably. :-)
>Additionally, as you noted, we can have real Zlib compression on top
>of that, compliments of Apache and Neon.
>
Yeah. And I'm wondering if compressing our delta stream is really worth
the effort.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by cm...@collab.net.
Chris Hecker <ch...@d6.com> writes:
> A delta/diff is different from a compressed stream. Sure, it's not
> sending the parts that didn't change, but the changes aren't
> compressed (and, as someone said, adds are the special case where it
> all changes), if I understand things correctly. Not that sending
> diffs isn't great, but you really want the compression to go both
> ways, especially since that's part of the reason one is supposed to
> use apache as the server from the manual ("wire compression for free").
Generally speaking, you're correct. But Subversion uses a binary
diffing format that not only sends just the differences, but even
those differences are transmitted in a gzip-like fashion. I forget
the stats someone figured out, but there definitely is some
compression going on. In other words, if your differences are that
you added the phrase "Chris is a pleasant programmer" in ten different
places through a file, the trasmitted data might list that string only
once, and then refer to it ten times.
Additionally, as you noted, we can have real Zlib compression on top
of that, compliments of Apache and Neon.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by "Martin v. Löwis" <ma...@v.loewis.de>.
Chris Hecker <ch...@d6.com> writes:
> Who can answer these questions authoritatively?
As a starting point, read the HTTP spec. Does it even allow clients to
send compressed input. Next, read the WebDAV spec. Does that allow to
send compressed input? By that time, you would know exactly how a
client would indicate whether it has compressed input. Search the Neon
source code for occurrences of the respective header, to find out
whether it ever emits that. Then check under what circumstances it
would do that.
SO: anybody can answer these questions authoritatively - in particular
you yourself.
Regards,
Martin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
--On Monday, June 23, 2003 1:44 PM -0700 Chris Hecker <ch...@d6.com> wrote:
>
>> > The above is all about server -> client stuff. He also wanted to know
>> > about whether stuff from the client to the server (e.g. commits) is ever
>> > gzip compressed. I assume the answer is "No", but maybe someone knows
>> > for sure.
>
> Since I'm new here, I assume the "someone" who would know reads this list?
Neon could be modified to send compressed request bodies. But, that's more
difficult because there is no 'handshake' to ensure that the server can handle
compressed request bodies. With responses, the client sends the
'Accept-Encoding: gzip' in the request, so there is a 'handshake' to ensure
that both sides can support deflate. So, compressing request bodies isn't a
realistic expectation to support with HTTP.
Using ra_svn with ssh tunneling with compression enabled would do both ways as
it blindly does compression on all data.
> Who can answer these questions authoritatively?
Someone on this list. ;-) -- justin
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Chris Hecker <ch...@d6.com>.
> > The above is all about server -> client stuff. He also wanted to know
> > about whether stuff from the client to the server (e.g. commits) is ever
> > gzip compressed. I assume the answer is "No", but maybe someone knows
> > for sure.
Since I'm new here, I assume the "someone" who would know reads this list?
>But I'm also not sure it matters... svn sends binary-diff data to the
>server when committing. This 'svndiff' data is already compressed via
>the vdelta algorithm. So in some sense, commits are *always*
>compressed.
A delta/diff is different from a compressed stream. Sure, it's not sending
the parts that didn't change, but the changes aren't compressed (and, as
someone said, adds are the special case where it all changes), if I
understand things correctly. Not that sending diffs isn't great, but you
really want the compression to go both ways, especially since that's part
of the reason one is supposed to use apache as the server from the manual
("wire compression for free").
A bit of reading on the apache site says that some DAV clients use client
-> server compression, so it should work in theory, but it seems like neon
would have to support that for subversion to use it transparently with
SetInputFilter, though. Hence the confusion.
And then there's the _subr compressed stream thing, which confuses me even
more.
Who can answer these questions authoritatively?
Thanks,
Chris
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Branko Čibej <br...@xbc.nu>.
Ben Collins-Sussman wrote:
>Michael Wood <mw...@its.uct.ac.za> writes:
>
>
>
>>The above is all about server -> client stuff. He also wanted to know
>>about whether stuff from the client to the server (e.g. commits) is ever
>>gzip compressed. I assume the answer is "No", but maybe someone knows
>>for sure.
>>
>>
>
>Hm, I'm nots sure I understand that part either.
>
>But I'm also not sure it matters... svn sends binary-diff data to the
>server when committing. This 'svndiff' data is already compressed via
>the vdelta algorithm. So in some sense, commits are *always*
>compressed.
>
>
Except when you'r sending new data for an added file.
In a sense, updates from the server are also always compressed;
checkouts are not.
--
Brane Čibej <br...@xbc.nu> http://www.xbc.nu/brane/
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Ben Collins-Sussman <su...@collab.net>.
Michael Wood <mw...@its.uct.ac.za> writes:
> The above is all about server -> client stuff. He also wanted to know
> about whether stuff from the client to the server (e.g. commits) is ever
> gzip compressed. I assume the answer is "No", but maybe someone knows
> for sure.
Hm, I'm nots sure I understand that part either.
But I'm also not sure it matters... svn sends binary-diff data to the
server when committing. This 'svndiff' data is already compressed via
the vdelta algorithm. So in some sense, commits are *always*
compressed.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Michael Wood <mw...@its.uct.ac.za>.
On Mon, Jun 23, 2003 at 06:39:39AM -0500, Ben Collins-Sussman wrote:
> Chris Hecker <ch...@d6.com> writes:
>
> > So, is compression done on the subversion side, or on the neon/http
> > side, and is it different for server->client vs. client->server?
>
> I could be very wrong, but here's how I understand the big picture:
>
[snip]
> * client makes an HTTP request to the server, and in the request
> headers indicates that it *is* able to understand deflated data.
... is able to understand deflated data coming from the server. (i.e.
this says nothing about whether the data the client sends to the server
is deflated.)
> * if the server has the deflate output-filter installed, and the
> client can deal with it, the server compresses the HTTP response
> body.
>
[snip]
> I'm sure someone will correct me if I'm wrong here...
The above is all about server -> client stuff. He also wanted to know
about whether stuff from the client to the server (e.g. commits) is ever
gzip compressed. I assume the answer is "No", but maybe someone knows
for sure.
--
Michael Wood <mw...@its.uct.ac.za>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: ra_dav compression question
Posted by Ben Collins-Sussman <su...@collab.net>.
Chris Hecker <ch...@d6.com> writes:
> So, is compression done on the subversion side, or on the neon/http
> side, and is it different for server->client vs. client->server?
I could be very wrong, but here's how I understand the big picture:
* on the client, neon is either compiled with or without zlib
support (-lz)
* client makes an HTTP request to the server, and in the request
headers indicates that it *is* able to understand deflated data.
* if the server has the deflate output-filter installed, and the
client can deal with it, the server compresses the HTTP response
body.
* if you're debugging the network trace, you need prevent this
compression. the best way to do this is to tell neon to *stop*
telling the server it can understand deflated data, by setting
'http-compression = no' in your ~/.subversion/servers file.
I'm sure someone will correct me if I'm wrong here...
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
ra_dav compression question
Posted by Chris Hecker <ch...@d6.com>.
I have a [possibly dumb] question about the compression done over http[s]
connections with subversion and apache. I searched and read the manual,
but still couldn't find an explanation. I browsed the source as well.
Compression from the server to the client seems to use mod_deflate, set up
with a SetOutputFilter DEFLATE in the svn Location section (according to
the subversion INSTALL). However, the docs don't say anything about using
the SetInputFilter DEFLATE option, which would seem to be necessary for
client to server compression, if it's done through the same
mechanism. Also, neon only seems to have a decompression facility, not
compression unless I missed it (even though the file is called
ne_compress.c). The libsvn_subr directory has a bidirectional compressed
stream object, to thicken the plot.
So, is compression done on the subversion side, or on the neon/http side,
and is it different for server->client vs. client->server?
Hope that question makes sense. Basically, I'm wondering why I need to
SetOutputFilter but not SetInputFilter, or if subversion and mod_dav_svn
are compressing themselves and then I'm double-compressing server->client
with mod_deflate.
Confused,
Chris
PS. Thanks for what looks to be a great revision system. I've got it
built and running over https on an Alpha server running Debian and the
build went fine. The only hitch was the mod_authz_svn got plopped in the
httpd.conf ahead of the mod_dav_svn line, which obviously didn't
work. Hopefully I will be able to replace CVS post haste. I worry a bit
about CPU/memory load on the server relative to CVS, since the machine is a
bit wimpy by today's standards.
PPS. Special bonus question that I haven't had time to look in the source
to answer: for the Win32 client, is there any way to use the
%HOME%\.subversion directory rather than \Documents and Settings\blah blah
blah\Subverion directory. Maybe if the latter doesn't exist but the former
does or something...it's much more convenient to use %HOME% if you're
already using it for a lot of other stuff.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: OT: Configuration management for stateful data
Posted by Daniel Patterson <da...@adaptiveinternational.com>.
On Wed, 2003-06-25 at 14:33, Bruce Atherton wrote:
> The upside to this is that it migrates schema changes to each environment
> synchronized with the release that depends on them. The downsides are that
> it is not easily reversible, and it provides no mechanism to return to the
> state at a particular release. I'm still looking for a good answer to that one.
I'm slowly coming to the conclusion that without "versioning" support
behind where the schema and data are being stored, it's always going
to be a process that is prone to human error (we're finding that people
are forgetting to run patches or running patches twice and deliberately
bypassing the safety checks (i.e. a "patch table" that lists applied
patches that the patches themselves check and update)).
All the versioning operations, such as diff, log and merge all have
use-cases in the data world as well.
Perhaps Subversion itself may find a nice market as a versioned database
:)
Subwiki may be the first example of a step towards this
(subwiki.tigris.org), it'd be interesting to attempt to map a relational
schema into a file tree....
daniel
--
Signature at: http://www.mel.au.adaptiveinternational.com/~danpat/signature.txt
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: OT: Configuration management for stateful data
Posted by Bruce Atherton <br...@callenish.com>.
At 02:58 PM 6/23/2003 +1000, Daniel Patterson wrote:
>Greetings all,
>
> I'm looking for advice on the best practises for configuration
> management of stateful data.
>
> This situation comes up in a few different places, but the
> best example (and the one I have right now) are database
> schema changes between software releases.
It is an interesting problem, and one for which I've been unable to find a
clean solution. It would be nice if you could version control the entire
schema. You would also want to version control the data in tables that
provides functionality to the program rather than just data, if you see
what I mean by the distinction.
What I do is version control and tag for release SQL patch files. Each
patch file has a comment at the top indicating the purpose of the change
along with the PR number. That is displayed to the release manager during
deployment, and they can choose whether to run the patch in each
environment, just as a double check that the change is intended to go with
the release.
The upside to this is that it migrates schema changes to each environment
synchronized with the release that depends on them. The downsides are that
it is not easily reversible, and it provides no mechanism to return to the
state at a particular release. I'm still looking for a good answer to that one.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org
Re: OT: Configuration management for stateful data
Posted by Ryan Rawson <ry...@netidea.com>.
Hi,
I work in ERP for a large company (that shall forever remain nameless),
and I think my experiences might benefit you a bit.
We use databases for our "stateful" data as you call it. I might call
it 'enterprise data' or 'business data'. When we do an upgrade to
database schemas usually its extending a table by adding a column, and
that can be done with a minimal outage (for live production systems)
and extending the database table. Without reloading the table (AFAIK,
I'm not a DBA). In any case, backups are your best friend when doing
any potentially destructive work on systems.
I think SVN has a role to play in management of the database "source" -
that is stored procedures (which suck) and schema specifications.
Management of the data itself is usually done with database backups,
save, restore, and all that good stuff.
I hope this helps.
Regards,
-ryan
On Sunday, June 22, 2003, at 09:58 PM, Daniel Patterson wrote:
> Greetings all,
>
> I'm looking for advice on the best practises for configuration
> management of stateful data.
>
> This situation comes up in a few different places, but the
> best example (and the one I have right now) are database
> schema changes between software releases.
>
> For stateless portions of the system (i.e. the application
> code), it's easy to overwrite, or drop/replace (in the case
> of database stored procedures). However, stateful portions
> (i.e. database tables with data in them, configuration files,
> any other stateful information that may undergo a schema change),
> the usual configuration management procedures don't apply.
> You can't drop 12 months of data from your database because
> you need to rename a relational table....
>
> A bit of searching around on google didn't reveal anything,
> but it's hard to know if I'm searching for the wrong thing or
> if nothing really exists.
>
> I've thought a bit about the process that Subversion takes
> (a dump into a known format followed by a reload), but in
> our situation, building a tool like that is impractical.
>
> Has anyone done any deep thinking about this topic and come
> up with any good patterns to follow?
>
> daniel
> --
> Signature at:
> http://www.mel.au.adaptiveinternational.com/~danpat/signature.txt
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
> For additional commands, e-mail: dev-help@subversion.tigris.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org