You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Keva-Slient <35...@qq.com> on 2018/01/03 02:37:20 UTC
inodes usage of svn server
inodes usage of my svn server grows up quickly as repositories commits increasing.
there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
is there any way storing all version files into one file? or any other way to solve this problem.
------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>
回复： inodes usage of svn server
Posted by Keva-Slient <35...@qq.com>.
thank you very much. that's exact what i want.
------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>
------------------ 原始邮件 ------------------
发件人: "Nico Kadel-Garcia";<nk...@gmail.com>;
发送时间: 2018年1月3日(星期三) 中午11:20
收件人: "Keva-Slient"<35...@qq.com>;
抄送: "users"<us...@subversion.apache.org>;
主题: Re: inodes usage of svn server
On Tue, Jan 2, 2018 at 9:37 PM, Keva-Slient <35...@qq.com> wrote:
>
> inodes usage of my svn server grows up quickly as repositories commits
> increasing.
> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
There is "svnadmin pack", which needs to run on the Subversion
repository server.
>
> is there any way storing all version files into one file? or any other way
> to solve this problem.
>
>
> ------------------
> .........
> Vanity...My favourite sin.
> --<<The Devil's Advocate>>
>
Re: inodes usage of svn server
Posted by Branko Čibej <br...@apache.org>.
On 03.01.2018 22:04, Bo Berglund wrote:
> On Wed, 3 Jan 2018 21:16:02 +0100, Branko ?ibej <br...@apache.org>
> wrote:
>
>> And there's your answer to the question of day to day performance of the
>> repository: Subversion also has to open many vs. just one file when its
>> reading historical revisions, so packing will definitely help to reduce
>> the number of directory lookups and file opens (the latter are
>> notoriously slow on Windows). Depending on usage patterns, the
>> performance boost may be significant.
> THat seems to be a valid cause for using the function then!
> I will look at my new converted repository in view of this info.
> Thanks!
It's safe to create a post-commit hook that runs 'svnadmin pack'
asynchrouously, or schedule a task that does that (hourly or daily,
depending on your needs). The operation is cheap if it has nothing to do.
-- Brane
Re: inodes usage of svn server
Posted by Bo Berglund <bo...@gmail.com>.
On Wed, 3 Jan 2018 21:16:02 +0100, Branko ?ibej <br...@apache.org>
wrote:
>And there's your answer to the question of day to day performance of the
>repository: Subversion also has to open many vs. just one file when its
>reading historical revisions, so packing will definitely help to reduce
>the number of directory lookups and file opens (the latter are
>notoriously slow on Windows). Depending on usage patterns, the
>performance boost may be significant.
THat seems to be a valid cause for using the function then!
I will look at my new converted repository in view of this info.
Thanks!
--
Bo Berglund
Developer in Sweden
Re: inodes usage of svn server
Posted by Branko Čibej <br...@apache.org>.
On 03.01.2018 21:13, Dave Huang wrote:
> On 1/3/2018 13:19, Nico Kadel-Garcia wrote:
>> NTFS also has limits on the maximum number of files for a filesystem.
>
> FWIW, that limit is 2^32 - 1 files, or approximately 4 billion (see
> Table 3.12 of https://technet.microsoft.com/en-us/library/cc938432.aspx)
>
>> There are also notable performance limits on having too many files in
>> a directory.
>
> I don't know enough about NTFS internals to say whether that's the
> case with NTFS or not, but in the context of this discussion, the
> default SVN shard size is 1000 revisions, which I don't think could be
> considered "too many files in a directory".
>
> While I don't have any actual numbers, my gut feeling is that packing
> the repo isn't really needed on NTFS in terms of day to day
> performance of clients using the repo, or in terms of filesystem
> limitations. That said, I do it anyway because backing up the repo
> does go faster when dealing with a couple dozen large files vs. tens
> of thousands of small files.
And there's your answer to the question of day to day performance of the
repository: Subversion also has to open many vs. just one file when its
reading historical revisions, so packing will definitely help to reduce
the number of directory lookups and file opens (the latter are
notoriously slow on Windows). Depending on usage patterns, the
performance boost may be significant.
-- Brane
Re: inodes usage of svn server
Posted by Dave Huang <kh...@azeotrope.org>.
On 1/3/2018 13:19, Nico Kadel-Garcia wrote:
> NTFS also has limits on the maximum number of files for a filesystem.
FWIW, that limit is 2^32 - 1 files, or approximately 4 billion (see
Table 3.12 of https://technet.microsoft.com/en-us/library/cc938432.aspx)
> There are also notable performance limits on having too many files in
> a directory.
I don't know enough about NTFS internals to say whether that's the case
with NTFS or not, but in the context of this discussion, the default SVN
shard size is 1000 revisions, which I don't think could be considered
"too many files in a directory".
While I don't have any actual numbers, my gut feeling is that packing
the repo isn't really needed on NTFS in terms of day to day performance
of clients using the repo, or in terms of filesystem limitations. That
said, I do it anyway because backing up the repo does go faster when
dealing with a couple dozen large files vs. tens of thousands of small
files.
--
Name: Dave Huang | Mammal, mammal / their names are called /
INet: khym@azeotrope.org | they raise a paw / the bat, the cat /
Telegram: @dahanc | dolphin and dog / koala bear and hog -- TMBG
Dahan: Hani G Y+C 42 Y++ L+++ W- C++ T++ A+ E+ S++ V++ F- Q+++ P+ B+ PA+
PL++
Re: inodes usage of svn server
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Wed, Jan 3, 2018 at 11:54 AM, Bo Berglund <bo...@gmail.com> wrote:
> On Wed, 3 Jan 2018 09:52:32 -0500, Nico Kadel-Garcia
> <nk...@gmail.com> wrote:
>
>>Please stop confusing "inode" with "disk space" It's more like you've
>>run out of boxes to put things in, i.e. inodes, than running out of
>>space to store store the boxes (i.e., disk space).
>
> Please do not confuse me with the original poster "Keva-Slient"....
> I merely stepped in to clarify if I might also have the problem, which
> he apparently is having.
> But I do not think so (I am on Windows Server 2016 with an NTFS file
> system).
Sorry about the quotation confusion.
NTFS also has limits on the maximum number of files for a filesystem.
There is no complete escape from the underlying limitations of
trade-offs of easy processing versus performance versus capacity that
lead to these limitations. There are also notable performance limits
on having too many files in a directory. We could have a long
discussion about the fundamentals: suffice to say that organizing a
bunch of associated bits into a single file, rather than scattering
many such bits around a filesystem and overpopulating a directory or a
filesystem is not your friend.
> --
> Bo Berglund
> Developer in Sweden
>
Re: inodes usage of svn server
Posted by Bo Berglund <bo...@gmail.com>.
On Wed, 3 Jan 2018 09:52:32 -0500, Nico Kadel-Garcia
<nk...@gmail.com> wrote:
>Please stop confusing "inode" with "disk space" It's more like you've
>run out of boxes to put things in, i.e. inodes, than running out of
>space to store store the boxes (i.e., disk space).
Please do not confuse me with the original poster "Keva-Slient"....
I merely stepped in to clarify if I might also have the problem, which
he apparently is having.
But I do not think so (I am on Windows Server 2016 with an NTFS file
system).
--
Bo Berglund
Developer in Sweden
Re: inodes usage of svn server
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Wed, Jan 3, 2018 at 1:50 AM, Bo Berglund <bo...@gmail.com> wrote:
> On Tue, 2 Jan 2018 22:20:13 -0500, Nico Kadel-Garcia
> <nk...@gmail.com> wrote:
>
>>On Tue, Jan 2, 2018 at 9:37 PM, Keva-Slient <35...@qq.com> wrote:
>>>
>>> inodes usage of my svn server grows up quickly as repositories commits
>>> increasing.
>>> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
>>
>>There is "svnadmin pack", which needs to run on the Subversion
>>repository server.
>>
>
> Is this a one-time cleanup operation or does it need to be executed
> regularly? I.e. does running "svnadmin pack" on the repo consolidate
> the many files into a smaller number of big files and set some
> repository property such that it will work that way in the future?
>
> This is what I found in svnbook 1.7 page 178:
>
> "By concatenating all the files of a completed
> shard into a single “pack” file and then removing the original
> per-revision files, svnadmin pack reduces the file count within a
> given shard down to just a single file."
>
> and
Disk space != inodes. You asked about inodes. File systems have a
maximum number of inodes, the maximum number of files directories,
links, etc. that they know how to count for that filesystem at the
time the filesystem is built. Storing information about files takes
space on a filesystem, sot it's notrmally pre-allocated when making
the filesystem. From your own description, you've run out of inodes
and need to free some up, or rebuild your environment to better handle
structures that handle many small files.
> "Repacking packed shards is legal, but will have no effect on the disk
> usage of the repository."
>
> What exactly is a "shard"?
> Is it one of the numerical directories each containing exactly 1000
> files?
It's one of the files in those directories.
> If so the "shard" storage on my repos seem to only cost at most 4%
> extra disk space compared to using a single file (comparing file sizes
> against disk usage).
> Or will compression of the "svnadmin pack" operation make the combined
> size of the files even smaller? I.e. it does not only save unused file
> allocation space but actually packs the content even better?
Please stop confusing "inode" with "disk space" It's more like you've
run out of boxes to put things in, i.e. inodes, than running out of
space to store store the boxes (i.e., disk space).
> --
> Bo Berglund
> Developer in Sweden
>
Re: inodes usage of svn server
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
Bo Berglund wrote on Wed, 03 Jan 2018 07:50 +0100:
> Is this a one-time cleanup operation or does it need to be executed
> regularly?
Regularly.
> I.e. does running "svnadmin pack" on the repo consolidate
> the many files into a smaller number of big files and set some
> repository property such that it will work that way in the future?
The former.
> This is what I found in svnbook 1.7 page 178:
>
> "By concatenating all the files of a completed
> shard into a single “pack” file and then removing the original
> per-revision files, svnadmin pack reduces the file count within a
> given shard down to just a single file."
>
> and
>
> "Repacking packed shards is legal, but will have no effect on the disk
> usage of the repository."
>
> What exactly is a "shard"?
> Is it one of the numerical directories each containing exactly 1000
> files?
>
Yes.
> If so the "shard" storage on my repos seem to only cost at most 4%
> extra disk space compared to using a single file (comparing file sizes
> against disk usage).
The concern was inode usage, not disk space usage. Packing reduces the
number of inodes by 99.8%.
> Or will compression of the "svnadmin pack" operation make the combined
> size of the files even smaller? I.e. it does not only save unused file
> allocation space but actually packs the content even better?
'svnadmin pack' neither recompresses nor redeltifies. It just
concatenates the files.
Re: inodes usage of svn server
Posted by Bo Berglund <bo...@gmail.com>.
On Tue, 2 Jan 2018 22:20:13 -0500, Nico Kadel-Garcia
<nk...@gmail.com> wrote:
>On Tue, Jan 2, 2018 at 9:37 PM, Keva-Slient <35...@qq.com> wrote:
>>
>> inodes usage of my svn server grows up quickly as repositories commits
>> increasing.
>> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
>
>There is "svnadmin pack", which needs to run on the Subversion
>repository server.
>
Is this a one-time cleanup operation or does it need to be executed
regularly? I.e. does running "svnadmin pack" on the repo consolidate
the many files into a smaller number of big files and set some
repository property such that it will work that way in the future?
This is what I found in svnbook 1.7 page 178:
"By concatenating all the files of a completed
shard into a single pack file and then removing the original
per-revision files, svnadmin pack reduces the file count within a
given shard down to just a single file."
and
"Repacking packed shards is legal, but will have no effect on the disk
usage of the repository."
What exactly is a "shard"?
Is it one of the numerical directories each containing exactly 1000
files?
If so the "shard" storage on my repos seem to only cost at most 4%
extra disk space compared to using a single file (comparing file sizes
against disk usage).
Or will compression of the "svnadmin pack" operation make the combined
size of the files even smaller? I.e. it does not only save unused file
allocation space but actually packs the content even better?
--
Bo Berglund
Developer in Sweden
svn over apache2 logging problem
Posted by Keva-Slient <35...@qq.com>.
here is my config in dav_svn.conf:
<Location /svn>
DAV svn
SVNParentPath /wns/data/svn
SVNListParentPath On
AuthType Basic
AuthName "Subversion Repository"
AuthUserFile /wns/etc/apache2/dav_svn.passwd
Require valid-user
</Location> #CustomLog /var/log/apache2/svn_logfile "%{%Y-%m-%d %T}t %u@%h %>s repo:%{SVN-REPOS-NAME}e %{SVN-ACTION}e (%B Bytes in %T Sec)" env=SVN-ACTION
CustomLog /var/log/apache2/svn_logfile "%t %u %{SVN-ACTION}e" env=SVN-ACTION
all of the log settings are not working.
it only creates log file. there is not any content written when i commit , checkout or update.
why is this?
------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>
Re: inodes usage of svn server
Posted by Nico Kadel-Garcia <nk...@gmail.com>.
On Tue, Jan 2, 2018 at 9:37 PM, Keva-Slient <35...@qq.com> wrote:
>
> inodes usage of my svn server grows up quickly as repositories commits
> increasing.
> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
There is "svnadmin pack", which needs to run on the Subversion
repository server.
>
> is there any way storing all version files into one file? or any other way
> to solve this problem.
>
>
> ------------------
> .........
> Vanity...My favourite sin.
> --<<The Devil's Advocate>>
>
Re: inodes usage of svn server
Posted by Keva-Slient <35...@qq.com>.
thank you very much. that's what i want
------------------
.........
Vanity...My favourite sin.
--<<The Devil's Advocate>>
------------------ Original ------------------
From: "Nathan Hartman";<ha...@gmail.com>;
Date: Wed, Jan 3, 2018 11:08 AM
To: "users"<us...@subversion.apache.org>;
Cc: "Keva-Slient"<35...@qq.com>;
Subject: Re: inodes usage of svn server
On Jan 2, 2018, at 9:37 PM, Keva-Slient <35...@qq.com> wrote:
>
>
> inodes usage of my svn server grows up quickly as repositories commits increasing.
> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
>
> is there any way storing all version files into one file? or any other way to solve this problem.
See "svnadmin pack" command in svn-book. I don't have the link handy unfortunately.
Re: inodes usage of svn server
Posted by Nathan Hartman <ha...@gmail.com>.
On Jan 2, 2018, at 9:37 PM, Keva-Slient <35...@qq.com> wrote:
>
>
> inodes usage of my svn server grows up quickly as repositories commits increasing.
> there are many reversion files in xx/db/revs/[0-9]+/[0-9]+.
>
> is there any way storing all version files into one file? or any other way to solve this problem.
See "svnadmin pack" command in svn-book. I don't have the link handy unfortunately.