You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Johan Corveleyn <jc...@gmail.com> on 2025/07/10 15:37:45 UTC
Re: Corrupt checkout, but 'svn status' and 'svn diff' refuse to admit it?
On Wed, Jul 9, 2025 at 9:53 PM EML <sa...@cyconix.com> wrote:
>
> I have a 2+ GB repo which I committed to today, to reach rev 357
> I then attempted to use a SQL database in the checkout, which turned out to be corrupt. When importing in MariaDB, it complained about a null char in the database
> The database was last committed at r335. I know it hasn't changed since r335, and 'svn log' shows r335 as the last commit, two months ago. I have imported the database on several occasions since then, without problems, when building a VM
> I then checked out the entire project at r335 to find out if the database has changed. If I do a standard diff of the database in r357 and r335 they are different. They are the same size, but the r357 version contains binary non-printing data, which is why MariaDB doesn't like it
> After the r357 checkin today, I modified one text file. I deleted this, and did an 'svn update' at the top level, to get a clean r357
> In the current checkout, 'svn status' shows no modifications. If I run 'svn diff -r335 my-database.sql' this reports no differences. So, svn thinks the new and old databases are identical, but the plain diff clearly shows they are different
> I then did a new complete checkout of the entire project at r357. The database in this new checkout is correct - it's identical to the r335 version
> The repo looks good - 'svnadmin dump' doesn't complain, anyway
> db/fs-type says fsfs
> Both the checkout and the repo are on the same machine, using WebDAV, version 1.14.1 (r1886195), Ubuntu 22.04
> There are some changes in the .svn directory between the "current" r357 and the just-checked-out r357
> This is a WordPress database. 'file' reports it as UTF-8 text, and 'wc -L' reports a maximum line length of 693319 characters. It doesn't have an svn:mime-type property.
>
> Conclusion: it's possible that a checkout can be corrupted in a way that bypasses detection by svn itself. Is this a reasonable conclusion, or have I missed something? Is there a maximum line length limit for a "text" file?
[ Moved dev@ to bcc so it is dropped on further replies, and added
users@ to cc. This is at first a question for the users list. ]
If I understand correctly you have one broken working copy in which
the contents of my-database.sql became corrupted somehow (containing
binary data, while it should be UTF-8 text and a valid database). In
the SVN repository that same file is still correct, and a new checkout
at r357 confirms that.
In general, the file in your $broken_workingcopy might have been
changed out-of-band, by some other tool, a virus, a disk failure, ...
but the main question is: why doesn't your $broken_workingcopy detect
that it's modified?
Is that a correct rephrasing of your question?
--
Johan
Re: Corrupt checkout, but 'svn status' and 'svn diff' refuse to admit it?
Posted by Branko Čibej <br...@apache.org>.
On 10. 7. 25 21:50, EML wrote:
>
> Hi -
>
> On 10/07/2025 19:36, Branko Čibej wrote:
>> On 10. 7. 25 17:37, Johan Corveleyn wrote:
>>> In general, the file in your $broken_workingcopy might have been
>>> changed out-of-band, by some other tool, a virus, a disk failure, ...
>>> but the main question is: why doesn't your $broken_workingcopy detect
>>> that it's modified?
>>>
>>> Is that a correct rephrasing of your question?
>>
>>
>> Subversion uses a shortcut when detecting changed files in the
>> working copy: if the file's size is the same size as recorded in the
>> metadata, and its modification time is the same as that recorded in
>> the metadata, svn assumes the file did not change.
>
> Interesting - so it's like the default rsync opts. I can (almost)
> understand 'svn status' doing this, but surely 'svn diff' should
> actually carry out a byte-by-byte comparison if the sizes are the same?
>
Nope. It's the same optimisation: if the file "hasn't changed", no diff
is performed.
Consider that to actually get this state -- change a file without
changing its mtime -- you have to do this on purpose; normal usage will
never get there. The most likely events for this to happen are either
memory or disk corruption, or an OS bug. In which case all (well,
"most") bets are off.
This is not perfectly ideal, but the alternative -- scanning the file
bytes -- would slow down the working copy by at least an order of
magnitude. We've been there and decided that the shortcut was the lesser
evil.
-- Brane
Re: Corrupt checkout, but 'svn status' and 'svn diff' refuse to admit it?
Posted by EML <sa...@cyconix.com>.
Hi -
On 10/07/2025 19:36, Branko Čibej wrote:
> On 10. 7. 25 17:37, Johan Corveleyn wrote:
>> In general, the file in your $broken_workingcopy might have been
>> changed out-of-band, by some other tool, a virus, a disk failure, ...
>> but the main question is: why doesn't your $broken_workingcopy detect
>> that it's modified?
>>
>> Is that a correct rephrasing of your question?
>
>
> Subversion uses a shortcut when detecting changed files in the working
> copy: if the file's size is the same size as recorded in the metadata,
> and its modification time is the same as that recorded in the
> metadata, svn assumes the file did not change.
Interesting - so it's like the default rsync opts. I can (almost)
understand 'svn status' doing this, but surely 'svn diff' should
actually carry out a byte-by-byte comparison if the sizes are the same?
This is Ubuntu 22.04, on an HPE ProLiant ML30 Gen30. Sadly, product
quality fell off a cliff when HP sold their PC line to HPE, so the root
cause was almost certainly a memory failure. The server's going in the
bin when I can find something else.
Re: Corrupt checkout, but 'svn status' and 'svn diff' refuse to admit it?
Posted by Branko Čibej <br...@apache.org>.
On 10. 7. 25 17:37, Johan Corveleyn wrote:
> On Wed, Jul 9, 2025 at 9:53 PM EML<sa...@cyconix.com> wrote:
>> I have a 2+ GB repo which I committed to today, to reach rev 357
>> I then attempted to use a SQL database in the checkout, which turned out to be corrupt. When importing in MariaDB, it complained about a null char in the database
>> The database was last committed at r335. I know it hasn't changed since r335, and 'svn log' shows r335 as the last commit, two months ago. I have imported the database on several occasions since then, without problems, when building a VM
>> I then checked out the entire project at r335 to find out if the database has changed. If I do a standard diff of the database in r357 and r335 they are different. They are the same size, but the r357 version contains binary non-printing data, which is why MariaDB doesn't like it
>> After the r357 checkin today, I modified one text file. I deleted this, and did an 'svn update' at the top level, to get a clean r357
>> In the current checkout, 'svn status' shows no modifications. If I run 'svn diff -r335 my-database.sql' this reports no differences. So, svn thinks the new and old databases are identical, but the plain diff clearly shows they are different
>> I then did a new complete checkout of the entire project at r357. The database in this new checkout is correct - it's identical to the r335 version
>> The repo looks good - 'svnadmin dump' doesn't complain, anyway
>> db/fs-type says fsfs
>> Both the checkout and the repo are on the same machine, using WebDAV, version 1.14.1 (r1886195), Ubuntu 22.04
>> There are some changes in the .svn directory between the "current" r357 and the just-checked-out r357
>> This is a WordPress database. 'file' reports it as UTF-8 text, and 'wc -L' reports a maximum line length of 693319 characters. It doesn't have ansvn:mime-type property.
>>
>> Conclusion: it's possible that a checkout can be corrupted in a way that bypasses detection by svn itself. Is this a reasonable conclusion, or have I missed something? Is there a maximum line length limit for a "text" file?
> [ Moved dev@ to bcc so it is dropped on further replies, and added
> users@ to cc. This is at first a question for the users list. ]
>
> If I understand correctly you have one broken working copy in which
> the contents of my-database.sql became corrupted somehow (containing
> binary data, while it should be UTF-8 text and a valid database). In
> the SVN repository that same file is still correct, and a new checkout
> at r357 confirms that.
>
> In general, the file in your $broken_workingcopy might have been
> changed out-of-band, by some other tool, a virus, a disk failure, ...
> but the main question is: why doesn't your $broken_workingcopy detect
> that it's modified?
>
> Is that a correct rephrasing of your question?
Subversion uses a shortcut when detecting changed files in the working
copy: if the file's size is the same size as recorded in the metadata,
and its modification time is the same as that recorded in the metadata,
svn assumes the file did not change.
You could try to change the file's modification time ('touch' on Unix,
but you don't say which platform you're on) and see if that affects
Subversion's output. Of course, this won't tell you anything about /why/
the file is corrupt.
Oh, by the way, you don't have a database committed to your repository,
from your description it's a dump/backup of the database contents.
-- Brane
Re: Corrupt checkout, but 'svn status' and 'svn diff' refuse to admit it?
Posted by EML <sa...@cyconix.com>.
On 10/07/2025 16:37, Johan Corveleyn wrote:
> On Wed, Jul 9, 2025 at 9:53 PM EML<sa...@cyconix.com> wrote:
>> (1) I have a 2+ GB repo which I committed to today, to reach rev 357
>> (2) I then attempted to use a SQL database in the checkout, which turned out to be corrupt. When importing in MariaDB, it complained about a null char in the database
>> (3) The database was last committed at r335. I know it hasn't changed since r335, and 'svn log' shows r335 as the last commit, two months ago. I have imported the database on several occasions since then, without problems, when building a VM
>> (4) I then checked out the entire project at r335 to find out if the database has changed. If I do a standard diff of the database in r357 and r335 they are different. They are the same size, but the r357 version contains binary non-printing data, which is why MariaDB doesn't like it
>> (5) After the r357 checkin today, I modified one text file. I deleted this, and did an 'svn update' at the top level, to get a clean r357
>> (6) In the current checkout, 'svn status' shows no modifications. If I run 'svn diff -r335 my-database.sql' this reports no differences. So, svn thinks the new and old databases are identical, but the plain diff clearly shows they are different
>> (7) I then did a new complete checkout of the entire project at r357. The database in this new checkout is correct - it's identical to the r335 version
>> (8) The repo looks good - 'svnadmin dump' doesn't complain, anyway
>> (9) db/fs-type says fsfs
>> (10) Both the checkout and the repo are on the same machine, using WebDAV, version 1.14.1 (r1886195), Ubuntu 22.04
>> (11) There are some changes in the .svn directory between the "current" r357 and the just-checked-out r357
>> (12) This is a WordPress database. 'file' reports it as UTF-8 text, and 'wc -L' reports a maximum line length of 693319 characters. It doesn't have an svn:mime-type property.
>>
>> Conclusion: it's possible that a checkout can be corrupted in a way that bypasses detection by svn itself. Is this a reasonable conclusion, or have I missed something? Is there a maximum line length limit for a "text" file?
> [ Moved dev@ to bcc so it is dropped on further replies, and added
> users@ to cc. This is at first a question for the users list. ]
>
> If I understand correctly you have one broken working copy in which
> the contents of my-database.sql became corrupted somehow (containing
> binary data, while it should be UTF-8 text and a valid database). In
> the SVN repository that same file is still correct, and a new checkout
> at r357 confirms that.
>
> In general, the file in your $broken_workingcopy might have been
> changed out-of-band, by some other tool, a virus, a disk failure, ...
> but the main question is: why doesn't your $broken_workingcopy detect
> that it's modified?
>
> Is that a correct rephrasing of your question?
Yes, that's right. I can live with a broken working copy, but only if
Subversion can tell me that it's broken.
Thanks. I've clarified some of the formatting above to make it obvious
that it was a list.
Re: Corrupt checkout, but 'svn status' and 'svn diff' refuse to admit it?
Posted by Branko Čibej <br...@apache.org>.
On 10. 7. 25 17:37, Johan Corveleyn wrote:
> On Wed, Jul 9, 2025 at 9:53 PM EML<sa...@cyconix.com> wrote:
>> I have a 2+ GB repo which I committed to today, to reach rev 357
>> I then attempted to use a SQL database in the checkout, which turned out to be corrupt. When importing in MariaDB, it complained about a null char in the database
>> The database was last committed at r335. I know it hasn't changed since r335, and 'svn log' shows r335 as the last commit, two months ago. I have imported the database on several occasions since then, without problems, when building a VM
>> I then checked out the entire project at r335 to find out if the database has changed. If I do a standard diff of the database in r357 and r335 they are different. They are the same size, but the r357 version contains binary non-printing data, which is why MariaDB doesn't like it
>> After the r357 checkin today, I modified one text file. I deleted this, and did an 'svn update' at the top level, to get a clean r357
>> In the current checkout, 'svn status' shows no modifications. If I run 'svn diff -r335 my-database.sql' this reports no differences. So, svn thinks the new and old databases are identical, but the plain diff clearly shows they are different
>> I then did a new complete checkout of the entire project at r357. The database in this new checkout is correct - it's identical to the r335 version
>> The repo looks good - 'svnadmin dump' doesn't complain, anyway
>> db/fs-type says fsfs
>> Both the checkout and the repo are on the same machine, using WebDAV, version 1.14.1 (r1886195), Ubuntu 22.04
>> There are some changes in the .svn directory between the "current" r357 and the just-checked-out r357
>> This is a WordPress database. 'file' reports it as UTF-8 text, and 'wc -L' reports a maximum line length of 693319 characters. It doesn't have ansvn:mime-type property.
>>
>> Conclusion: it's possible that a checkout can be corrupted in a way that bypasses detection by svn itself. Is this a reasonable conclusion, or have I missed something? Is there a maximum line length limit for a "text" file?
> [ Moved dev@ to bcc so it is dropped on further replies, and added
> users@ to cc. This is at first a question for the users list. ]
>
> If I understand correctly you have one broken working copy in which
> the contents of my-database.sql became corrupted somehow (containing
> binary data, while it should be UTF-8 text and a valid database). In
> the SVN repository that same file is still correct, and a new checkout
> at r357 confirms that.
>
> In general, the file in your $broken_workingcopy might have been
> changed out-of-band, by some other tool, a virus, a disk failure, ...
> but the main question is: why doesn't your $broken_workingcopy detect
> that it's modified?
>
> Is that a correct rephrasing of your question?
Subversion uses a shortcut when detecting changed files in the working
copy: if the file's size is the same size as recorded in the metadata,
and its modification time is the same as that recorded in the metadata,
svn assumes the file did not change.
You could try to change the file's modification time ('touch' on Unix,
but you don't say which platform you're on) and see if that affects
Subversion's output. Of course, this won't tell you anything about /why/
the file is corrupt.
Oh, by the way, you don't have a database committed to your repository,
from your description it's a dump/backup of the database contents.
-- Brane