svn commit: r1803274 - /subversion/site/publish/faq.html

[jc...@apache.org]

Author: jcorvel
Date: Fri Jul 28 13:55:36 2017
New Revision: 1803274

URL: http://svn.apache.org/viewvc?rev=1803274&view=rev
Log:
* site/publish/faq.html
  (dumpload): Rework and extend this section, with a more complex procedure
   for dump+load with minimal downtime.
   
  This is based on the following post to the users list:
   
  http://mail-archives.apache.org/mod_mbox/subversion-users/201606.mbox/%3CCAB84uBUTuzgYL7z1ZhiW%2BnoVeVYe6R78VcpXer2VqQsAkq2%2B-g%40mail.gmail.com%3E
  Date: Thu, 16 Jun 2016 10:49:43 GMT
  From: Johan Corveleyn
  To: Subversion users list
  Subject: Re: Invalid character in hex checksum?
  Message ID: <CAB84uBUTuzgYL7z1ZhiW%2BnoVeVYe6R78VcpXer2VqQsAkq2%2B-g%40mail.gmail.com>


Modified:
    subversion/site/publish/faq.html

Modified: subversion/site/publish/faq.html
URL: http://svn.apache.org/viewvc/subversion/site/publish/faq.html?rev=1803274&r1=1803273&r2=1803274&view=diff
==============================================================================
--- subversion/site/publish/faq.html (original)
+++ subversion/site/publish/faq.html Fri Jul 28 13:55:36 2017
@@ -1395,11 +1395,35 @@ see <a href="https://issues.apache.org/j
 </h3>
 
 <p>Subversion's repository database schema has changed occasionally
-during development. Old repositories, created with a pre-1.0
-development version of Subversion, may require the following operation
-when upgrading.  If a schema change happens between Subversion
-releases X and Y, then repository administrators upgrading to Y must
-do the following: </p>
+during development.  To take advantage of new features, you may have
+to dump and load the repository to recreate the back-end database.
+However, most upgrades of Subversion do <i>not</i> involve a
+dump and load.  When one is required, the release notes and the
+CHANGES file for the new version will carry prominent notices about
+it.  If you don't see such a notice, then there has been no schema
+change, and no dump/load is necessary.</p>
+
+<p>An alternative to dump/load is using
+<a href="http://svnbook.red-bean.com/en/1.8/svn.reposadmin.maint.html#svn.reposadmin.maint.replication"
+>svnsync</a> to replicate the repository into a new one.  This is a bit
+slower, but is more flexible, and has some extra normalization-features
+which are not (yet) available with dump/load (svnsync normalizes properties
+to LF line-endings on the fly and has a --source-prop-encoding option
+to convert them to UTF-8, which is required in newer repository formats
+--- see below for how to handle this with dump/load).</p>
+
+<p><b>Note: </b>Both dump/load and svnsync <b>only cover the repository
+database, not the repository hook scripts, configuration files and
+locks</b>.  These need to be copied over manually from source to target
+(see below in the "complex procedure").
+If you need to copy the complete repository, without rebuilding the back-end
+database, <a href="http://svnbook.red-bean.com/en/1.8/svn.reposadmin.maint.html#svn.reposadmin.maint.backup"
+>svnadmin hotcopy</a> may be a better option.</p>
+
+<p>For small repositories that can afford some downtime, this is a 
+simple dump/load procedure to upgrade from Subversion version X to Y (
+see below for a more complex procedure with minimal downtime for larger
+repositories):</p>
 
 <ol>
  <li>Shut down svnserve, Apache, and anything else that might be
@@ -1441,17 +1465,150 @@ do the following: </p>
  </li>
 </ol>
 
+<p>For larger repositories, where you want to minimize the maintenance window,
+a slightly more complex procedure can be used.  The trick is to
+dump+load to a new location, while the old repository is still
+accessible (for checkouts and commits).  After this is done (can take
+hours, or even days, weeks) you note the last revision which was
+loaded (or check the revision number with 'svnlook youngest
+newrepos'), and start another dump+load where you dump with
+'--incremental -rNEXTREV:HEAD' (where NEXTREV is the next revision
+that needs to be dumped).  You can iterate over this as long as you keep the old repository open
+... At the end you make the original repository inaccessible for a
+couple of minutes, while you enable the new one (Caveat: if you move
+your new repos in the same disk location as the old one, and you use
+Apache httpd to serve it, make sure you restart httpd to reset its
+caches).</p>
+
+<p>Tip: for a large repo I strongly suggest building the new repository
+(the target of your 'svnadmin load') on very fast storage, even
+ramdisk if possible, and running 'svnadmin pack' while on fast storage
+(and copy it over to the final disk afterwards).  It's
+the 'svnadmin load' part that is very time-consuming right now (this
+will probably be much improved in svn 1.10, with the
+--no-flush-to-disk option for 'svnadmin load').</p>
+
+<p>Your commands will look like the following:</p>
+<ol>
+<li><b>
+<tt>svnadmin create NEWREPOS</tt>
+</b>
+<br/>(maybe create it on a ramdisk)
+</li>
+
+<li>
+If you have custom hook scripts in <b><tt>OLDREPOS/hooks</tt></b> (all files
+not ending in .tmpl, as those are the default templates), review them,
+and copy them over to <b><tt>NEWREPOS/hooks</tt></b>. Check the new templates
+corresponding to your custom hook scripts to see if there are new options 
+and comments (you might want to copy the newer comments from the template
+into your custom hook script, to keep it up to date). Make sure the files are
+not changed anymore until the end of the entire procedure (or carry over
+additional changes at the end).
+</li>
+
+<li>
+Review and copy config files from <b><tt>OLDREPOS/conf</tt></b> to
+<b><tt>NEWREPOS/conf</tt></b> (here too, take a look at the new
+"default config files" to see if there are new interesting options or
+comments). Make sure the files are not changed anymore until the end of the
+entire procedure (or carry over additional changes at the end).
+</li>
+
+<li><b>
+<tt>svnadmin dump -M 1024 OLDREPOS | svnadmin load -M 1024 NEWREPOS</tt>
+</b>
+<br/>(initial dump+load; you might want to pass -q to dump and/or load
+to make it more quiet)
+<br/>(the -M 1024 gives the process 1024 MB extra ram for caching) 
+</li>
+
+<li><b>
+<tt>svnlook youngest NEWREPOS</tt>
+</b>
+<br/>(last revision that was loaded -> NEXTREV is this last revision + 1)
+</li>
+
+<li><b>
+<tt>svnadmin dump --incremental -rNEXTREV:HEAD -M 1024 | svnadmin load
+-M 1024 NEWRPOS</tt>
+</b>
+</li>
+
+<li>
+Make OLDREPOS read-only or completely unavailable <b><-- start of maintenance window</b>
+</li>
+
+<li>
+Possibly repeat the incremental dump+load of steps 5 and 6 (if new commits
+happened after you started 6).
+</li>
+
+<li>
+Copy locks from <b><tt>OLDREPOS/db/locks</tt></b> to
+<b><tt>NEWREPOS/db/locks</tt></b>.  Something like 'cp -rp SOURCE TARGET'
+works fine for this. <b>Note: this step may take a couple of minutes during
+your maintenance window, depending on the size of the directory tree
+and the speed of the disk(s).  If you want to make sure, test this in 
+advance (but make sure the source repository is readonly when you do 
+the final copy / sync, otherwise the locks might be changed after your
+copy).</b>
+</li>
+
+<li>
+Put NEWREPOS online <b><-- end of maintenance window</b>
+</li>
+</ol>
+
+<p>Some things to watch out for:</p>
+
+<ul>
+<li>You might run into:
+<pre>
+svnadmin: E125005: Invalid property value found in dumpstream; consider repairing the
+source or using --bypass-prop-validation while loading.
+
+svnadmin: E125005: Cannot accept non-LF line endings in 'svn:log' property
+</pre>
+
+This means the svn:log message of the revision has non-LF line endings (these
+were accepted by older servers, but no longer as of Subversion 1.6).
+You can accept this minor corruption by adding --bypass-prop-validation
+to your 'svnadmin load' command (you can always repair this later in the 
+new repository).  Or you can try to repair this in the source repository
+before executing the dump+load (since svn:log is a <i>revision property</i>
+it can easily be fixed without "rewriting history").
+</li>
+
+<li>You might run into:
+<pre>
+svnadmin: E125005: Invalid property value found in dumpstream; consider repairing the
+source or using --bypass-prop-validation while loading.
+
+svnadmin: E125005: Cannot accept non-LF line endings in 'svn:ignore' property
+</pre>
+
+This is more difficult to repair, because 'svn:ignore' is not a
+revision property (like svn:log, which can be manipulated with
+svnadmin setrevprop), but a <i>versioned property</i> (so it's part of
+history).  Again, you can ignore this with --bypass-prop-validation.
+But since this is a corruption "in history", this can only be repaired
+with a dump+load, so this might be a good time to try and fix this (or
+you'll run into this again in the future).  To repair it you can use
+a tool like <a href="https://github.com/jwiegley/svndumptool">svndumptool</a>.
+But it only works on dump <i>files</i>, not as part of a pipe.  So a possible
+way to go about it is: dump that single
+(corrupt) revision to a file, repair it ('svndumptool.py eolfix-prop
+svn:ignore svn.dump svn.dump.repaired'), load that single dumpfile,
+and then continue with a new "piped" command (like step (6) above). 
+</li>
+</ul>
+
 <p> See
 <a href="http://svnbook.red-bean.com/nightly/en/svn.reposadmin.maint.html#svn.reposadmin.maint.migrate"
 >this section of the Subversion book</a> for more details on dumping and
 loading.</p>
 
-<p> <b>Note</b>: Most upgrades of Subversion do <i>not</i> involve a
-dump and load.  When one is required, the release announcement and the
-CHANGES file for the new version will carry prominent notices about
-it.  If you don't see such a notice, then there has been no schema
-change, and no dump/load is necessary. </p>
-
 </div>