You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by William Muriithi <wi...@gmail.com> on 2026/08/05 22:46:34 UTC
Re: Could this be SVN shard directory logic bug?
Hi guys
This is just for closure
The main reason was RestrictSUIDSGID=yes, I had to override it for apache
to allow chmod fro svn
/etc/systemd/system/httpd.service.d/override.conf
[Service]
ReadWritePaths=/var/repos/svn
UMask=0002
RestrictSUIDSGID=no
On Tue, 21 Jul 2026 at 09:36, Barry <ba...@barrys-emacs.org> wrote:
>
>
> On 25 May 2026, at 19:45, William Muriithi <wi...@gmail.com>
> wrote:
>
>
> Hi Daniel
>
> If you look at one of your existing working copies (run for example svn
>> info from the command line), what is the highest revision number you have
>> there?
>>
> Have the same revision # on the client too
>
>
> william@william-ryzen:~/Documents/application$ svn info
> Path: .
> Working Copy Root Path: /home/william/Documents/application
> URL: https://carbon.example.com/svn/application
> Relative URL: ^/
> Repository Root: https://carbon.example.com/svn/application
> Repository UUID: 02a4eb3e-e1fa-4ead-9f6a-c97b0409a8a7
> Revision: 269999
> Node Kind: directory
> Schedule: normal
> Last Changed Author: nathan
> Last Changed Rev: 269999
>
>
> I hit a bug like this with a repo that jammed on 1999.
> I used something to force in r2000 and then everything started working
> again.
>
> I suspect I use root and committed a trivial change use file:// url to the
> repo.
> But it was a while ago and I did not keep notes.
>
> Barty
>
> Last Changed Date: 2026-05-11 15:54:29 -0400 (Mon, 11 May 2026)
>
> william@william-ryzen:~/Documents/application$
>
>
>>
>> I have a feeling that something has caused the 270 directory to
>> disappear, thus your repository is now in an inconsistent state. I'm not
>> sure if you can recover except restoring a backup. If you can run svnadmin
>> dump and then svnadmin load to a new repository, you can probably get a
>> working repository with the first 269999 revisions, but you of course loose
>> everything later than that.
>>
>
> Possibly, I am open to that being the case. Is there a way to be
> confident this is what happened?
>
> - I have two systems - the backup which was using svnsync is also in the
> same state. I suspect having a replica doesn't change much as cron was
> scheduled to run every 5 minutes, so if the directory 270 was created and
> destroyed within 5 minutes, the replica wouldn't have picked the change
> [root@boron ~]# cat /usr/local/bin/svnsync.sh
> /usr/bin/svnsync sync file:///var/repos/svn/projects
> - The last svndump restore leaves the repo in the same state. Wouldn't
> the repo dump have flagged that something is off? Wouldn't the restore the
> restore fail when it try to replay the entry 270080 that SVN seems to
> assume exit?
> - The svnadmin verify pass all the commits up to 269999
>
> What else can we do to figure out where the corruption is?
>
>>
>> To answer the question about maximum revision number: The ASF main
>> repository (https://svn.apache.org/repos/asf) is currently at revision
>> 1934607, so over 7 times your repository. I'm not sure about size but I
>> believe it is quite huge.
>>
>>
> Ok, so there exit repo bigger than this then, thanks.
>
>
> Regards,
> William
>
>> Kind regards,
>> Daniel
>>
>>
>>
>> Den mån 25 maj 2026 kl 19:13 skrev William Muriithi <
>> william.muriithi@gmail.com>:
>>
>>> Hi guys and girls
>>>
>>>
>>> Something I missed to mention. Originally, the file contents looked like
>>> this:-
>>>
>>> [root@repos ~]# cat /var/repos/svn/projects/db/current
>>> 269999
>>> [root@repos ~]# cat /var/repos/svn/projects/db/txn-current
>>> 5sbz
>>>
>>> These files seem to be important on how SVN makes the decision of which
>>> directory to dump/look for a file. So I looked around and google result
>>> suggested that the last file may be incorrect.
>>>
>>> So I changed it to:-
>>>
>>> [root@repos ~]# cat /var/repos/svn/projects/db/txn-current
>>> 5r3j
>>>
>>> MOVE forward:
>>> I then created the files strace point as SVN access points. All below
>>> commands were executed as user apache
>>>
>>> mkdir /var/repos/svn/projects/db/revs/270
>>> cd /var/repos/svn/projects/db/revs/270
>>> touch 270000 270016 270080 270336
>>> svnadmin recover /var/repos/svn/projects/
>>>
>>> When I tried committing a new, I got a new error:-
>>>
>>> svnadmin: E160004: Revision 270336 has a revs file but no revprops file
>>>
>>> MOVE backward:
>>> [root@repos ~]# cat /var/repos/svn/projects/db/current
>>> 269999
>>> [root@repos ~]# cat /var/repos/svn/projects/db/txn-current
>>> 5r3j
>>>
>>> svnadmin recover /var/repos/svn/projects/ (As apache user) - It run
>>> successfully.
>>>
>>> Then we went back to SVN client and got the old error
>>>
>>> Basically, we can't seem to move it back or forward.
>>>
>>> Regards,
>>> William
>>>
>>>
>>>
>>> On Mon, 25 May 2026 at 12:24, William Muriithi <
>>> william.muriithi@gmail.com> wrote:
>>>
>>>> Hi guys,
>>>>
>>>> [root@repos ~]# rpm -qa | grep subversion
>>>> subversion-libs-1.14.5-3.el10.x86_64_v2
>>>> subversion-1.14.5-3.el10.x86_64_v2
>>>> subversion-tools-1.14.5-3.el10.x86_64_v2
>>>> [root@repos ~]#
>>>> [root@repos ~]#
>>>> [root@repos ~]# cat /etc/redhat-release
>>>> AlmaLinux release 10.1 (Heliotrope Lion)
>>>> [root@repos ~]#
>>>> [root@repos ~]# rpm -qa | grep svn
>>>> mod_dav_svn-1.14.5-3.el10.x86_64_v2
>>>>
>>>>
>>>> I think I have hit a bug. We have a relatively large SVN repository -
>>>> About 2TB uncompressed. 2 weeks ago, someone noticed that we can't commit
>>>> any more. When we looked at it, it looked like a permission issue.
>>>>
>>>> This is the error we are getting from the SVN client.
>>>>
>>>> svn: E000001: Commit failed (details follow):
>>>> svn: E000001: Can't set permissions on
>>>> '/var/repos/svn/application/db/revs/270'
>>>> william@william-ryzen:~/Documents/application$
>>>> users@subversion.apache.org
>>>>
>>>> So we first attempted to "FIX" permissions. Didn't work. Then, when
>>>> reading, I noticed opening up the permission is not a good idea as I can
>>>> see from strace that svn does try to narrow down the file permissions. So
>>>> we decided to restore a fresh to remove all the chmod we had done to the
>>>> files and directory.
>>>>
>>>> This time, when restoring, we created the repo using an apache
>>>> account. We also loaded the dump file using the svn account. This removed
>>>> all possibility this is a file permission issue. The repo was restored
>>>> successfully, BUT the problem remained.
>>>>
>>>> We looked at the file limit too and to make sure that isn't causing any
>>>> issues, we changed it to unlimited for the apache user. The problem
>>>> remained.
>>>>
>>>> It's at this point we attempted to use strace to see what svn is
>>>> doing. Oddly, it seemed to be looking for files from directory 270, which
>>>> didn't exist at the time. We came to the conclusion that was what was
>>>> happening because these came up when one attempts to commit.
>>>>
>>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336",
>>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336",
>>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080",
>>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080",
>>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016",
>>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016",
>>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000",
>>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>>>> openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000",
>>>> O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
>>>>
>>>> Note, the directory 270 didn't exist. According to google, this can
>>>> happen due to either a hook or CI committing too often creating
>>>> corruption. We can't seem to clear the corruption though. We are also a
>>>> bit skeptical of its corruption as it should have cleared on a freshly
>>>> restored repository.
>>>>
>>>>
>>>> So over the weekend, we attempted to separate the projects on the repo
>>>> to individual repos. This is to reduce the size of the repo as it could
>>>> have been causing the issue. We however kept the revision numbers as they
>>>> are in documentations and tags. However, on the now small repo, with only
>>>> 187 GB in size, but with the last successful commit # 269999, we still
>>>> see the problem.
>>>>
>>>> This now looks more like a software bug than a setup issue. We have
>>>> "yum updated" the system in an attempt to make sure we have all the bug
>>>> fixes out there but nothing works.
>>>>
>>>> Would anyone know if there is a transaction limit in the current SVN
>>>> code? Anyone have a repository with more than 270000 revisions out there
>>>> please? Any pointers to something we may have overlooked so far?
>>>> Something else, the svndump is cleanly at 2 TB, any chance there may be a
>>>> limit of how big a repo glow?? This was our first thought but since we
>>>> have created smaller repos and the issue remains, it now looks far more
>>>> like a sharding logic error to us.
>>>>
>>>>
>>>> Regards,
>>>> William
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
RE: Could this be SVN shard directory logic bug?
Posted by "falco philmarks.net" <fa...@philmarks.net>.
Good afternoon,
It turns out that William’s general approach solved my problem as well.
My issue with “Can't write '/var/svn/NetBeansProjects/db/txn-current' atomically” has now gone away.
After hunting around on my Ubuntu 26.04 server, the relevant file turned out to be:
/lib/systemd/system/apache2.service
After changing
RestrictSUIDSGID= yes to RestrictSUIDSGID= no
commits and other changes to the repository worked as expected.
Furthermore, examining the same config file on my older Ubuntu 24.04 server showed that the version of apache2.service distributed with 24.04 was much simpler than the one in 26.04, and omitted the RestrictSUIDGUID line entirely. This seems like a convincing reason for the different behavior between the two servers.
I too had suspected that the values in the “current” and “txn-current” files might have been the source of the problem, but it now seems they were OK all along.
Similarly, I did some digging into logs and configurations re AppArmor, but I didn’t find anything relevant.
As a relative newbie to Ubuntu and to this list, there is not one chance in a million I would have been able to find and fix this problem without help, so many thanks to Jeff, William, and Daniel.
Phil Marks
From: William Muriithi <wi...@gmail.com>
Sent: Wednesday, August 5, 2026 6:47 PM
To: users@subversion.apache.org
Cc: Andreas Stieger <an...@gmx.de>
Subject: Re: Could this be SVN shard directory logic bug?
Hi guys
This is just for closure
The main reason was RestrictSUIDSGID=yes, I had to override it for apache to allow chmod fro svn
/etc/systemd/system/httpd.service.d/override.conf
[Service]
ReadWritePaths=/var/repos/svn
UMask=0002
RestrictSUIDSGID=no
On Tue, 21 Jul 2026 at 09:36, Barry <ba...@barrys-emacs.org>> wrote:
On 25 May 2026, at 19:45, William Muriithi <wi...@gmail.com>> wrote:
Hi Daniel
If you look at one of your existing working copies (run for example svn info from the command line), what is the highest revision number you have there?
Have the same revision # on the client too
william@william-ryzen:~/Documents/application$ svn info
Path: .
Working Copy Root Path: /home/william/Documents/application
URL: https://carbon.example.com/svn/application
Relative URL: ^/
Repository Root: https://carbon.example.com/svn/application
Repository UUID: 02a4eb3e-e1fa-4ead-9f6a-c97b0409a8a7
Revision: 269999
Node Kind: directory
Schedule: normal
Last Changed Author: nathan
Last Changed Rev: 269999
I hit a bug like this with a repo that jammed on 1999.
I used something to force in r2000 and then everything started working again.
I suspect I use root and committed a trivial change use file:// url to the repo.
But it was a while ago and I did not keep notes.
Barty
Last Changed Date: 2026-05-11 15:54:29 -0400 (Mon, 11 May 2026)
william@william-ryzen:~/Documents/application$
I have a feeling that something has caused the 270 directory to disappear, thus your repository is now in an inconsistent state. I'm not sure if you can recover except restoring a backup. If you can run svnadmin dump and then svnadmin load to a new repository, you can probably get a working repository with the first 269999 revisions, but you of course loose everything later than that.
Possibly, I am open to that being the case. Is there a way to be confident this is what happened?
- I have two systems - the backup which was using svnsync is also in the same state. I suspect having a replica doesn't change much as cron was scheduled to run every 5 minutes, so if the directory 270 was created and destroyed within 5 minutes, the replica wouldn't have picked the change
[root@boron ~]# cat /usr/local/bin/svnsync.sh
/usr/bin/svnsync sync file:///var/repos/svn/projects
- The last svndump restore leaves the repo in the same state. Wouldn't the repo dump have flagged that something is off? Wouldn't the restore the restore fail when it try to replay the entry 270080 that SVN seems to assume exit?
- The svnadmin verify pass all the commits up to 269999
What else can we do to figure out where the corruption is?
To answer the question about maximum revision number: The ASF main repository (https://svn.apache.org/repos/asf) is currently at revision 1934607, so over 7 times your repository. I'm not sure about size but I believe it is quite huge.
Ok, so there exit repo bigger than this then, thanks.
Regards,
William
Kind regards,
Daniel
Den mån 25 maj 2026 kl 19:13 skrev William Muriithi <wi...@gmail.com>>:
Hi guys and girls
Something I missed to mention. Originally, the file contents looked like this:-
[root@repos ~]# cat /var/repos/svn/projects/db/current
269999
[root@repos ~]# cat /var/repos/svn/projects/db/txn-current
5sbz
These files seem to be important on how SVN makes the decision of which directory to dump/look for a file. So I looked around and google result suggested that the last file may be incorrect.
So I changed it to:-
[root@repos ~]# cat /var/repos/svn/projects/db/txn-current
5r3j
MOVE forward:
I then created the files strace point as SVN access points. All below commands were executed as user apache
mkdir /var/repos/svn/projects/db/revs/270
cd /var/repos/svn/projects/db/revs/270
touch 270000 270016 270080 270336
svnadmin recover /var/repos/svn/projects/
When I tried committing a new, I got a new error:-
svnadmin: E160004: Revision 270336 has a revs file but no revprops file
MOVE backward:
[root@repos ~]# cat /var/repos/svn/projects/db/current
269999
[root@repos ~]# cat /var/repos/svn/projects/db/txn-current
5r3j
svnadmin recover /var/repos/svn/projects/ (As apache user) - It run successfully.
Then we went back to SVN client and got the old error
Basically, we can't seem to move it back or forward.
Regards,
William
On Mon, 25 May 2026 at 12:24, William Muriithi <wi...@gmail.com>> wrote:
Hi guys,
[root@repos ~]# rpm -qa | grep subversion
subversion-libs-1.14.5-3.el10.x86_64_v2
subversion-1.14.5-3.el10.x86_64_v2
subversion-tools-1.14.5-3.el10.x86_64_v2
[root@repos ~]#
[root@repos ~]#
[root@repos ~]# cat /etc/redhat-release
AlmaLinux release 10.1 (Heliotrope Lion)
[root@repos ~]#
[root@repos ~]# rpm -qa | grep svn
mod_dav_svn-1.14.5-3.el10.x86_64_v2
I think I have hit a bug. We have a relatively large SVN repository - About 2TB uncompressed. 2 weeks ago, someone noticed that we can't commit any more. When we looked at it, it looked like a permission issue.
This is the error we are getting from the SVN client.
svn: E000001: Commit failed (details follow):
svn: E000001: Can't set permissions on '/var/repos/svn/application/db/revs/270'
william@william-ryzen:~/Documents/application$ users@subversion.apache.org<ma...@subversion.apache.org>
So we first attempted to "FIX" permissions. Didn't work. Then, when reading, I noticed opening up the permission is not a good idea as I can see from strace that svn does try to narrow down the file permissions. So we decided to restore a fresh to remove all the chmod we had done to the files and directory.
This time, when restoring, we created the repo using an apache account. We also loaded the dump file using the svn account. This removed all possibility this is a file permission issue. The repo was restored successfully, BUT the problem remained.
We looked at the file limit too and to make sure that isn't causing any issues, we changed it to unlimited for the apache user. The problem remained.
It's at this point we attempted to use strace to see what svn is doing. Oddly, it seemed to be looking for files from directory 270, which didn't exist at the time. We came to the conclusion that was what was happening because these came up when one attempts to commit.
openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270336", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270080", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270016", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/var/repos/svn/projects/db/revs/270/270000", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
Note, the directory 270 didn't exist. According to google, this can happen due to either a hook or CI committing too often creating corruption. We can't seem to clear the corruption though. We are also a bit skeptical of its corruption as it should have cleared on a freshly restored repository.
So over the weekend, we attempted to separate the projects on the repo to individual repos. This is to reduce the size of the repo as it could have been causing the issue. We however kept the revision numbers as they are in documentations and tags. However, on the now small repo, with only 187 GB in size, but with the last successful commit # 269999, we still see the problem.
This now looks more like a software bug than a setup issue. We have "yum updated" the system in an attempt to make sure we have all the bug fixes out there but nothing works.
Would anyone know if there is a transaction limit in the current SVN code? Anyone have a repository with more than 270000 revisions out there please? Any pointers to something we may have overlooked so far? Something else, the svndump is cleanly at 2 TB, any chance there may be a limit of how big a repo glow?? This was our first thought but since we have created smaller repos and the issue remains, it now looks far more like a sharding logic error to us.
Regards,
William