You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by Philip Martin <ph...@codematters.co.uk> on 2014/03/24 18:27:15 UTC
svn_io_sleep_for_timestamps not sleeping long enough
I'm seeing that checkout does not always sleep for long enough for
timestamp-based change detection to be reliable:
svnadmin create repo
svn -mm import repo/format file://`pwd`/repo/f
svn co file://`pwd`/repo wc
echo x > wc/f
svn st wc
For some checkouts status shows the modification of wc/f, for other
checkouts it does not.
The problem is that my filesystem has about 4ms timestamp resolution
while svn_io_sleep_for_timestamps() only sleeps for 1ms. I can
determine the resolution experimentally using:
echo x > f && stat f | grep Modify && echo y > f && stat f | grep Modify
often that prints the same timestamp twice, but when the timestamps
differ they typically differ by about 4ms:
Modify: 2014-03-24 16:40:17.566569826 +0000
Modify: 2014-03-24 16:40:17.570569785 +0000
In order to avoid missing changes I need to change
svn_io_sleep_for_timestamps:
Index: subversion/libsvn_subr/io.c
===================================================================
--- subversion/libsvn_subr/io.c (revision 1580884)
+++ subversion/libsvn_subr/io.c (working copy)
@@ -1315,7 +1315,7 @@ svn_io_sleep_for_timestamps(const char *path, apr_
/* Sleep for at least 1 millisecond.
(t < 1000 will be round to 0 in apr) */
- apr_sleep(1000);
+ apr_sleep(4000);
return;
}
I'm running a standard 64-bit Debian/stable kernel with working copies
on an ext4 filesystem on a local SSD. I believe the 4ms is related to
the kernel's CONFIG_HZ=250 setting (it also has CONFIG_NO_HZ=y). Now I
could build a custom kernel and increase the resolution but I think
Subversion should support what is essentially a standard Linux setup. I
think we should increase the sleep to at least 10ms on the basis that
Linux kernels built with CONFIG_HZ=100 are not unusual. This does
penalise systems with true nanosecond resolution as they will sleep 9ms
more than necessary.
I've also got an ext3 filesystem on a spinning disk but that doesn't
show the problem because timestamps are always whole seconds and so
svn_io_sleep_for_timestamps uses a different code path.
--
Philip
Re: svn_io_sleep_for_timestamps not sleeping long enough
Posted by Philip Martin <ph...@wandisco.com>.
Branko Čibej <br...@wandisco.com> writes:
> I would guess that the easiest way to solve this would be to
> periodically touch and stat the WC root dir until its mtime changes.
> Unfortunatesly, sleep-for-timestamps has not the slightest idea where
> the WC root might be ...
svn_io_sleep_for_timestamps does get an optional wc path, and checkout
does supply a path. Implementing "touch" for a directory in a portable
way might be tricky.
--
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*
Re: svn_io_sleep_for_timestamps not sleeping long enough
Posted by Branko Čibej <br...@wandisco.com>.
On 24.03.2014 19:47, Philip Martin wrote:
> Julian Foad <ju...@btopenworld.com> writes:
>
>> Philip Martin wrote:
>>
>>> Julian Foad writes:
>>>>> Modify: 2014-03-24 16:40:17.566569826 +0000
>>>>> Modify: 2014-03-24 16:40:17.570569785 +0000
>>>> Is it possible to query the system clock resolution at run time? That
>>>> would seem best.
>>> I believe the system clock resolution is different from the filesystem
>>> resolution.
>> Correct. We need to know both to perform an optimal sleep.
> I don't think the clock resolution tells us much. When I try:
>
> echo x>f && date +'%F %T.%N' && echo y>g && date +'%F %T.%N' && stat f g | grep Modify
>
> I sometimes get output like:
>
> 2014-03-24 18:38:41.600661633
> 2014-03-24 18:38:41.601409267
> Modify: 2014-03-24 18:38:41.596646220 +0000
> Modify: 2014-03-24 18:38:41.596646220 +0000
>
> So the files f and g have the same timestamp even though date between
> the two writes shows two different, later, times. In this case any
> check comparing 'now' to the first file would indicate that 'now' is
> later than first file, but the second file still gets the same timestamp
> as the first file.
I would guess that the easiest way to solve this would be to
periodically touch and stat the WC root dir until its mtime changes.
Unfortunatesly, sleep-for-timestamps has not the slightest idea where
the WC root might be ...
This would still leave us open to the weird case where parts of the
working copy are bind-mounted to a different filesystem, but I'm quite
happy to ignore that edge case; solving it would require storing the
equivalent of a device number for every directory in the working copy.
-- Brane
--
Branko Čibej | Director of Subversion
WANdisco // Non-Stop Data
e. brane@wandisco.com
Re: svn_io_sleep_for_timestamps not sleeping long enough
Posted by Philip Martin <ph...@wandisco.com>.
Julian Foad <ju...@btopenworld.com> writes:
> Philip Martin wrote:
>
>> Julian Foad writes:
>>>> Modify: 2014-03-24 16:40:17.566569826 +0000
>>>> Modify: 2014-03-24 16:40:17.570569785 +0000
>>>
>>> Is it possible to query the system clock resolution at run time? That
>>> would seem best.
>>
>> I believe the system clock resolution is different from the filesystem
>> resolution.
>
> Correct. We need to know both to perform an optimal sleep.
I don't think the clock resolution tells us much. When I try:
echo x>f && date +'%F %T.%N' && echo y>g && date +'%F %T.%N' && stat f g | grep Modify
I sometimes get output like:
2014-03-24 18:38:41.600661633
2014-03-24 18:38:41.601409267
Modify: 2014-03-24 18:38:41.596646220 +0000
Modify: 2014-03-24 18:38:41.596646220 +0000
So the files f and g have the same timestamp even though date between
the two writes shows two different, later, times. In this case any
check comparing 'now' to the first file would indicate that 'now' is
later than first file, but the second file still gets the same timestamp
as the first file.
--
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*
Re: svn_io_sleep_for_timestamps not sleeping long enough
Posted by Julian Foad <ju...@btopenworld.com>.
Philip Martin wrote:
> Julian Foad writes:
>>> Modify: 2014-03-24 16:40:17.566569826 +0000
>>> Modify: 2014-03-24 16:40:17.570569785 +0000
>>
>> Is it possible to query the system clock resolution at run time? That
>> would seem best.
>
> I believe the system clock resolution is different from the filesystem
> resolution.
Correct. We need to know both to perform an optimal sleep.
- Julian
Re: svn_io_sleep_for_timestamps not sleeping long enough
Posted by Philip Martin <ph...@wandisco.com>.
Julian Foad <ju...@btopenworld.com> writes:
>> Modify: 2014-03-24 16:40:17.566569826 +0000
>> Modify: 2014-03-24 16:40:17.570569785 +0000
>
> Is it possible to query the system clock resolution at run time? That
> would seem best.
I believe the system clock resolution is different from the filesystem
resolution.
--
Philip Martin | Subversion Committer
WANdisco // *Non-Stop Data*
Re: svn_io_sleep_for_timestamps not sleeping long enough
Posted by Justin Erenkrantz <ju...@erenkrantz.com>.
On Mon, Mar 24, 2014 at 1:35 PM, Julian Foad <ju...@btopenworld.com> wrote:
> But, compared with the present 1 ms that's inadequate on many systems and the 1 second that's too much on some but not enough on others, I would be happy with simply increasing the 1 ms to 10 ms for a "quick fix" if we don't want to invest any more effort.
+1 to 10ms...or even 100ms. -- justin
Re: svn_io_sleep_for_timestamps not sleeping long enough
Posted by Julian Foad <ju...@btopenworld.com>.
--
Join WANdisco's free daily demo sessions on Scaling Subversion for the Enterprise
<http://www.wandisco.com/training/webinars>
----- Original Message -----
> From: Philip Martin <ph...@codematters.co.uk>
> To: dev@subversion.apache.org
> Cc:
> Sent: Monday, 24 March 2014, 17:27
> Subject: svn_io_sleep_for_timestamps not sleeping long enough
>
> I'm seeing that checkout does not always sleep for long enough for
> timestamp-based change detection to be reliable:
>
> svnadmin create repo
> svn -mm import repo/format file://`pwd`/repo/f
> svn co file://`pwd`/repo wc
> echo x > wc/f
> svn st wc
>
> For some checkouts status shows the modification of wc/f, for other
> checkouts it does not.
>
> The problem is that my filesystem has about 4ms timestamp resolution
> while svn_io_sleep_for_timestamps() only sleeps for 1ms. I can
> determine the resolution experimentally using:
>
> echo x > f && stat f | grep Modify && echo y > f
> && stat f | grep Modify
>
> often that prints the same timestamp twice, but when the timestamps
> differ they typically differ by about 4ms:
>
> Modify: 2014-03-24 16:40:17.566569826 +0000
> Modify: 2014-03-24 16:40:17.570569785 +0000
>
> In order to avoid missing changes I need to change
> svn_io_sleep_for_timestamps:
>
> Index: subversion/libsvn_subr/io.c
> ===================================================================
> --- subversion/libsvn_subr/io.c (revision 1580884)
> +++ subversion/libsvn_subr/io.c (working copy)
> @@ -1315,7 +1315,7 @@ svn_io_sleep_for_timestamps(const char *path, apr_
>
> /* Sleep for at least 1 millisecond.
> (t < 1000 will be round to 0 in apr) */
> - apr_sleep(1000);
> + apr_sleep(4000);
>
> return;
> }
>
> I'm running a standard 64-bit Debian/stable kernel with working copies
> on an ext4 filesystem on a local SSD. I believe the 4ms is related to
> the kernel's CONFIG_HZ=250 setting (it also has CONFIG_NO_HZ=y). Now I
> could build a custom kernel and increase the resolution but I think
> Subversion should support what is essentially a standard Linux setup. I
> think we should increase the sleep to at least 10ms on the basis that
> Linux kernels built with CONFIG_HZ=100 are not unusual. This does
> penalise systems with true nanosecond resolution as they will sleep 9ms
> more than necessary.
>
> I've also got an ext3 filesystem on a spinning disk but that doesn't
> show the problem because timestamps are always whole seconds and so
> svn_io_sleep_for_timestamps uses a different code path.
Is it possible to query the system clock resolution at run time? That would seem best.
But, compared with the present 1 ms that's inadequate on many systems and the 1 second that's too much on some but not enough on others, I would be happy with simply increasing the 1 ms to 10 ms for a "quick fix" if we don't want to invest any more effort.
- Julian