You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@subversion.apache.org by Thuan Seah Tan <th...@fmod.com> on 2019/06/04 04:55:29 UTC
svn status and info slowness when multiple files are passed as args
Wondering if anyone has experienced slowness when running commandline svn
if multiple files are passed to svn status or svn info?
For example:
svn status directory/file1.txt directory/file2.txt directory/file3.txt
In my tests, the output is generated at about the rate of 1 file per second
when doing so. Running svn status on the entire directory recursively turns
out to be much faster.
Thuan
Re: svn status and info slowness when multiple files are passed as args
Posted by Thuan Seah Tan <th...@fmod.com>.
Thanks for the info. I did more tests and can confirm the issue appears to
be a IPv6 vs IPv4 issue.
I enabled the --prefer-ipv6 option on the svnserve command and running
commands on checked out files that point to a domain name repository root
were significantly faster. Alternatively relocating the project root from
domain name to explicit IPv4 also helped.
On Fri, Jun 28, 2019 at 4:37 AM Johan Corveleyn <jc...@gmail.com> wrote:
> On Wed, Jun 26, 2019 at 9:33 AM Thuan Seah Tan <th...@fmod.com> wrote:
> >
> > Apologies for the delay on this issue. I did more testing and the
> following was my findings:
> >
> > svn info -r HEAD "svn://mysvnserver/1.10/test.txt" "svn://<server
> name>/1.10/test2.txt" <-- slow
> > svn info -r HEAD "svn://192.168.1.123/1.10/test.txt" "svn://<IPv4
> address of server>/1.10/test2.txt" <-- fast
> >
> > If I am not using the svn protocol and just passing in the file path on
> disk, depending on how the files were checked out:
> >
> > if checked out using Tortoise SVN and specifying the repository server
> as "svn://192.168.1.123":
> > svn info -r HEAD "C:/1.10/test.txt" "C:/1.10/test2.txt" <-- fast
> >
> > if checked out using Tortoise SVN and specifying the address of the
> server as "svn://mysvnserver":
> > svn info -r HEAD "C:/1.10/test.txt" "C:/1.10/test2.txt" <-- slow
> >
> > Wondering if using the server name defaults to IPv6. I suppose that's up
> to the router's configuration? When checking out files, is there something
> added to the .svn folder that retains the knowledge of whether a file was
> checked out using ipv4 or server name?
> >
>
> Ok, so it's clearly a problem with the client first trying the IPv6
> address, to which svnserve doesn't respond in your case. Now, I don't
> know a lot myself about how to deal with that situation (we don't use
> svnserve, we're accessing our svn repository via https). Perhaps
> someone else on this list has some experience, and can help a bit?
>
> Just a couple of thoughts:
> - The fact that svn://mysvnserver first defaults to the IPv6 address
> is entirely up to network configuration, I guess. Not sure if it's the
> router, local DNS server, Windows networking configuration, ... as I
> said, I don't know much in that area. But perhaps that's the easiest
> "fix" for you: fix the network configuration so clients get connected
> to the IPv4 address by default.
>
> - The .svn folder certainly retains knowledge of the exact URL it was
> checked out from. It's visible when you run 'svn info' (showing the
> working copy url). You can change that in an existing working copy by
> running 'svn relocate' (see 'svn help relocate' for help), so you can
> "connect" it with another URL pointing to the same repository.
>
> - Perhaps that thread I pointed to earlier contains some more
> interesting information about what else you can do.
> https://svn.haxx.se/users/archive-2018-06/0000.shtml
> (for instance, I see things about starting up a second instance of
> svnserve with the -6 option for listening on the IPv6 address)
>
> HTH,
> --
> Johan
>
Re: svn status and info slowness when multiple files are passed as args
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Jun 26, 2019 at 9:33 AM Thuan Seah Tan <th...@fmod.com> wrote:
>
> Apologies for the delay on this issue. I did more testing and the following was my findings:
>
> svn info -r HEAD "svn://mysvnserver/1.10/test.txt" "svn://<server name>/1.10/test2.txt" <-- slow
> svn info -r HEAD "svn://192.168.1.123/1.10/test.txt" "svn://<IPv4 address of server>/1.10/test2.txt" <-- fast
>
> If I am not using the svn protocol and just passing in the file path on disk, depending on how the files were checked out:
>
> if checked out using Tortoise SVN and specifying the repository server as "svn://192.168.1.123":
> svn info -r HEAD "C:/1.10/test.txt" "C:/1.10/test2.txt" <-- fast
>
> if checked out using Tortoise SVN and specifying the address of the server as "svn://mysvnserver":
> svn info -r HEAD "C:/1.10/test.txt" "C:/1.10/test2.txt" <-- slow
>
> Wondering if using the server name defaults to IPv6. I suppose that's up to the router's configuration? When checking out files, is there something added to the .svn folder that retains the knowledge of whether a file was checked out using ipv4 or server name?
>
Ok, so it's clearly a problem with the client first trying the IPv6
address, to which svnserve doesn't respond in your case. Now, I don't
know a lot myself about how to deal with that situation (we don't use
svnserve, we're accessing our svn repository via https). Perhaps
someone else on this list has some experience, and can help a bit?
Just a couple of thoughts:
- The fact that svn://mysvnserver first defaults to the IPv6 address
is entirely up to network configuration, I guess. Not sure if it's the
router, local DNS server, Windows networking configuration, ... as I
said, I don't know much in that area. But perhaps that's the easiest
"fix" for you: fix the network configuration so clients get connected
to the IPv4 address by default.
- The .svn folder certainly retains knowledge of the exact URL it was
checked out from. It's visible when you run 'svn info' (showing the
working copy url). You can change that in an existing working copy by
running 'svn relocate' (see 'svn help relocate' for help), so you can
"connect" it with another URL pointing to the same repository.
- Perhaps that thread I pointed to earlier contains some more
interesting information about what else you can do.
https://svn.haxx.se/users/archive-2018-06/0000.shtml
(for instance, I see things about starting up a second instance of
svnserve with the -6 option for listening on the IPv6 address)
HTH,
--
Johan
Re: svn status and info slowness when multiple files are passed as args
Posted by Thuan Seah Tan <th...@fmod.com>.
Apologies for the delay on this issue. I did more testing and the following
was my findings:
svn info -r HEAD "svn://mysvnserver/1.10/test.txt" "svn://<server
name>/1.10/test2.txt" <-- slow
svn info -r HEAD "svn://192.168.1.123/1.10/test.txt" "svn://<IPv4 address
of server>/1.10/test2.txt" <-- fast
If I am not using the svn protocol and just passing in the file path on
disk, depending on how the files were checked out:
if checked out using Tortoise SVN and specifying the repository server as
"svn://192.168.1.123":
svn info -r HEAD "C:/1.10/test.txt" "C:/1.10/test2.txt" <-- fast
if checked out using Tortoise SVN and specifying the address of the server
as "svn://mysvnserver":
svn info -r HEAD "C:/1.10/test.txt" "C:/1.10/test2.txt" <-- slow
Wondering if using the server name defaults to IPv6. I suppose that's up to
the router's configuration? When checking out files, is there something
added to the .svn folder that retains the knowledge of whether a file was
checked out using ipv4 or server name?
On Fri, Jun 14, 2019 at 12:13 AM Johan Corveleyn <jc...@gmail.com> wrote:
> On Thu, Jun 13, 2019 at 1:59 PM Thorsten Schöning <ts...@am-soft.de>
> wrote:
> >
> > Guten Tag Thuan Seah Tan,
> > am Donnerstag, 13. Juni 2019 um 09:25 schrieben Sie:
> >
> > > [...]I also tried running 'info -r HEAD' on files
> > > that are checked out on the PC that the server was running, and it
> > > is just as slow. Both the url and repository root fields started with
> "svn://localhost".
> >
> > As you are running Windows, disabling all kinds of AV-software at
> > least for the directories belonging to your SVN-repos would be the
> > first thing I'm trying. If that doesn't change a thing, use Process
> > Monitor to see where the slowness comes from. That reports exactly
> > which I/O happens where and how long it takes.
> >
> > https://docs.microsoft.com/en-us/sysinternals/downloads/procmon
> >
> https://docs.microsoft.com/en-us/sysinternals/downloads/sysinternals-suite
> >
> > Mit freundlichen Grüßen,
> >
> > Thorsten Schöning
>
> Hm, I don't think the problem is "local IO slowdown" (like with
> antivirus). That wouldn't explain "svn info -r HEAD -R directory"
> taking 1-2 seconds (in the initial report by the OP).
>
> @Thuan: has it always been that slow?
> My intuition tells me to take a look at IPv4 vs. IPv6 problems. See
> for example this thread on this mailinglist:
> https://svn.haxx.se/users/archive-2018-06/0000.shtml
>
> Also, in response to your question:
> >> I am using Tortoise SVN 1.12.0 r1857323. When you say it could be
> optimized in the client, I take it that is up to the team maintaining the
> project (e.g. Tortoise SVN, Visual SVN, etc) and I should report the issue
> to them? Or is there some base client code used by these implementations?
>
> Yes, there is "base client code" shared by all these implementations:
> TortoiseSVN, Visual SVN, commandline SVN, ... they all share the same
> underlying svn libraries that are maintained by this project, Apache
> Subversion (of which you've reached the users mailinglist).
>
> As for "reporting the issue to the team maintaining the project":
> there is not really a dedicated "team" waiting to work on issues.
> There are project members of the Apache Subversion project (some of
> them are also reading this mailinglist -- I am one of them). Those
> project members are just individuals like yourself, sometimes working
> on things they care about (for themselves or for their employers) ...
> such is the nature of FOSS. So reporting an issue or suggesting an
> improvement will not magically make it happen. On the other hand, we
> very much appreciate clear reports of issues or suggestions -- those
> are definitely valuable contributions.
>
> In other words: yes, it could be a good idea to officially write this
> down into an issue in the issue tracker [1] (but we're still a bit
> fuzzy on the details, I think we still need some further discussion /
> troubleshooting), but to make expectations clear: reporting it does
> not magically make it happen :-).
>
> [1] http://subversion.apache.org/reporting-issues.html
>
> Thanks,
> --
> Johan
>
Re: svn status and info slowness when multiple files are passed as args
Posted by Johan Corveleyn <jc...@gmail.com>.
On Thu, Jun 13, 2019 at 1:59 PM Thorsten Schöning <ts...@am-soft.de> wrote:
>
> Guten Tag Thuan Seah Tan,
> am Donnerstag, 13. Juni 2019 um 09:25 schrieben Sie:
>
> > [...]I also tried running 'info -r HEAD' on files
> > that are checked out on the PC that the server was running, and it
> > is just as slow. Both the url and repository root fields started with "svn://localhost".
>
> As you are running Windows, disabling all kinds of AV-software at
> least for the directories belonging to your SVN-repos would be the
> first thing I'm trying. If that doesn't change a thing, use Process
> Monitor to see where the slowness comes from. That reports exactly
> which I/O happens where and how long it takes.
>
> https://docs.microsoft.com/en-us/sysinternals/downloads/procmon
> https://docs.microsoft.com/en-us/sysinternals/downloads/sysinternals-suite
>
> Mit freundlichen Grüßen,
>
> Thorsten Schöning
Hm, I don't think the problem is "local IO slowdown" (like with
antivirus). That wouldn't explain "svn info -r HEAD -R directory"
taking 1-2 seconds (in the initial report by the OP).
@Thuan: has it always been that slow?
My intuition tells me to take a look at IPv4 vs. IPv6 problems. See
for example this thread on this mailinglist:
https://svn.haxx.se/users/archive-2018-06/0000.shtml
Also, in response to your question:
>> I am using Tortoise SVN 1.12.0 r1857323. When you say it could be optimized in the client, I take it that is up to the team maintaining the project (e.g. Tortoise SVN, Visual SVN, etc) and I should report the issue to them? Or is there some base client code used by these implementations?
Yes, there is "base client code" shared by all these implementations:
TortoiseSVN, Visual SVN, commandline SVN, ... they all share the same
underlying svn libraries that are maintained by this project, Apache
Subversion (of which you've reached the users mailinglist).
As for "reporting the issue to the team maintaining the project":
there is not really a dedicated "team" waiting to work on issues.
There are project members of the Apache Subversion project (some of
them are also reading this mailinglist -- I am one of them). Those
project members are just individuals like yourself, sometimes working
on things they care about (for themselves or for their employers) ...
such is the nature of FOSS. So reporting an issue or suggesting an
improvement will not magically make it happen. On the other hand, we
very much appreciate clear reports of issues or suggestions -- those
are definitely valuable contributions.
In other words: yes, it could be a good idea to officially write this
down into an issue in the issue tracker [1] (but we're still a bit
fuzzy on the details, I think we still need some further discussion /
troubleshooting), but to make expectations clear: reporting it does
not magically make it happen :-).
[1] http://subversion.apache.org/reporting-issues.html
Thanks,
--
Johan
Re: svn status and info slowness when multiple files are passed as args
Posted by Thorsten Schöning <ts...@am-soft.de>.
Guten Tag Thuan Seah Tan,
am Donnerstag, 13. Juni 2019 um 09:25 schrieben Sie:
> [...]I also tried running 'info -r HEAD' on files
> that are checked out on the PC that the server was running, and it
> is just as slow. Both the url and repository root fields started with "svn://localhost".
As you are running Windows, disabling all kinds of AV-software at
least for the directories belonging to your SVN-repos would be the
first thing I'm trying. If that doesn't change a thing, use Process
Monitor to see where the slowness comes from. That reports exactly
which I/O happens where and how long it takes.
https://docs.microsoft.com/en-us/sysinternals/downloads/procmon
https://docs.microsoft.com/en-us/sysinternals/downloads/sysinternals-suite
Mit freundlichen Grüßen,
Thorsten Schöning
--
Thorsten Schöning E-Mail: Thorsten.Schoening@AM-SoFT.de
AM-SoFT IT-Systeme http://www.AM-SoFT.de/
Telefon...........05151- 9468- 55
Fax...............05151- 9468- 88
Mobil..............0178-8 9468- 04
AM-SoFT GmbH IT-Systeme, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 207 694 - Geschäftsführer: Andreas Muchow
Re: svn status and info slowness when multiple files are passed as args
Posted by Thuan Seah Tan <th...@fmod.com>.
Hi Johan,
I am using Tortoise SVN 1.12.0 r1857323. When you say it could be optimized
in the client, I take it that is up to the team maintaining the project
(e.g. Tortoise SVN, Visual SVN, etc) and I should report the issue to them?
Or is there some base client code used by these implementations?
I ran 'svn info --show-item url' as requested and the url showed the svn://
protocol. I also tried running 'info -r HEAD' on files that are checked out
on the PC that the server was running, and it is just as slow. Both the url
and repository root fields started with "svn://localhost".
On Wed, Jun 5, 2019 at 8:05 PM Johan Corveleyn <jc...@gmail.com> wrote:
> On Wed, Jun 5, 2019 at 4:15 AM Thuan Seah Tan <th...@fmod.com> wrote:
> >
> > Thanks for the response. I did more testing on the 1.12.0 server, and it
> seems it's only those command options that I think would require querying
> the server is experiencing the slow down:
> >
> > e.g.
> > svn status --show-updates directory/file1.txt directory/file2.txt
> directory/file3.txt <-- this took about 3 seconds and seems to scale
> according to the number of files as it outputs "Status against revision"
> for each file.
> > svn status --show-updates directory <-- this took about 1-2 seconds but
> only output "Status against revision" once.
> >
> > svn info -r HEAD directory/file1.txt directory/file2.txt
> directory/file3.txt <-- this took about 3 seconds and seems to scale
> according to the number of files and display info for each file at the rate
> of 1 file per second
> > svn info -r HEAD -R directory <-- this took about 1-2 seconds even
> though the entire directory has 17 files and just outputs info for 17 files
> in one hit
> >
> > The server is on another machine in my local network and both running
> Windows 10 Pro. Not entirely sure the filesystem you are referring to, but
> the drive with the repository is running NTFS.
>
> Is your client also version 1.12.0? Running 'svn --version' on the
> client will tell.
> It's just to eliminate that this was possibly optimized on the client
> side in recent versions.
>
> A couple of other things come to mind:
>
> - It's possible that 'svn status --show-updates X Y Z' opens
> (sequentially) 3 separate connections / sessions to the server,
> instead of only 1. Same for your 'svn info -r HEAD' example. That's
> something that could possibly be optimized in the client.
>
> - How come it even takes 1-2 seconds for a single 'status
> --show-updates' or 'info -r HEAD' request? That's extremely slow,
> especially since you're saying it's all on a local network. Is it fast
> if you create a working copy with a file:/// URL directly on the
> server, and perform those commands there (it should be fast)? What
> protocol are client and server using? Running 'svn info --show-item
> url' on your working copy should tell. If it's https://, maybe there
> is some problem / slowdown when opening a new SSL socket? Or
> performing LDAP authentication on the server side. If it's svn://,
> maybe there is a problem with IP v6 vs. IP v4.
>
> --
> Johan
>
Re: svn status and info slowness when multiple files are passed as args
Posted by Johan Corveleyn <jc...@gmail.com>.
On Wed, Jun 5, 2019 at 4:15 AM Thuan Seah Tan <th...@fmod.com> wrote:
>
> Thanks for the response. I did more testing on the 1.12.0 server, and it seems it's only those command options that I think would require querying the server is experiencing the slow down:
>
> e.g.
> svn status --show-updates directory/file1.txt directory/file2.txt directory/file3.txt <-- this took about 3 seconds and seems to scale according to the number of files as it outputs "Status against revision" for each file.
> svn status --show-updates directory <-- this took about 1-2 seconds but only output "Status against revision" once.
>
> svn info -r HEAD directory/file1.txt directory/file2.txt directory/file3.txt <-- this took about 3 seconds and seems to scale according to the number of files and display info for each file at the rate of 1 file per second
> svn info -r HEAD -R directory <-- this took about 1-2 seconds even though the entire directory has 17 files and just outputs info for 17 files in one hit
>
> The server is on another machine in my local network and both running Windows 10 Pro. Not entirely sure the filesystem you are referring to, but the drive with the repository is running NTFS.
Is your client also version 1.12.0? Running 'svn --version' on the
client will tell.
It's just to eliminate that this was possibly optimized on the client
side in recent versions.
A couple of other things come to mind:
- It's possible that 'svn status --show-updates X Y Z' opens
(sequentially) 3 separate connections / sessions to the server,
instead of only 1. Same for your 'svn info -r HEAD' example. That's
something that could possibly be optimized in the client.
- How come it even takes 1-2 seconds for a single 'status
--show-updates' or 'info -r HEAD' request? That's extremely slow,
especially since you're saying it's all on a local network. Is it fast
if you create a working copy with a file:/// URL directly on the
server, and perform those commands there (it should be fast)? What
protocol are client and server using? Running 'svn info --show-item
url' on your working copy should tell. If it's https://, maybe there
is some problem / slowdown when opening a new SSL socket? Or
performing LDAP authentication on the server side. If it's svn://,
maybe there is a problem with IP v6 vs. IP v4.
--
Johan
Re: svn status and info slowness when multiple files are passed as args
Posted by Thuan Seah Tan <th...@fmod.com>.
Thanks for the response. I did more testing on the 1.12.0 server, and it
seems it's only those command options that I think would require querying
the server is experiencing the slow down:
e.g.
svn status --show-updates directory/file1.txt directory/file2.txt
directory/file3.txt <-- this took about 3 seconds and seems to scale
according to the number of files as it outputs "Status against revision"
for each file.
svn status --show-updates directory <-- this took about 1-2 seconds but
only output "Status against revision" once.
svn info -r HEAD directory/file1.txt directory/file2.txt
directory/file3.txt <-- this took about 3 seconds and seems to scale
according to the number of files and display info for each file at the rate
of 1 file per second
svn info -r HEAD -R directory <-- this took about 1-2 seconds even though
the entire directory has 17 files and just outputs info for 17 files in one
hit
The server is on another machine in my local network and both running
Windows 10 Pro. Not entirely sure the filesystem you are referring to, but
the drive with the repository is running NTFS.
On Tue, Jun 4, 2019 at 6:18 PM Johan Corveleyn <jc...@gmail.com> wrote:
> On Tue, Jun 4, 2019 at 7:35 AM Thuan Seah Tan <th...@fmod.com> wrote:
> >
> > Wondering if anyone has experienced slowness when running commandline
> svn if multiple files are passed to svn status or svn info?
> >
> > For example:
> > svn status directory/file1.txt directory/file2.txt directory/file3.txt
> >
> > In my tests, the output is generated at about the rate of 1 file per
> second when doing so. Running svn status on the entire directory
> recursively turns out to be much faster.
>
> Doesn't ring a bell with me. Can you give some more context?
> - Which OS?
> - Which filesystem? Is it local or a network filesystem?
> - Which version of SVN? If not the latest (1.12.0), can you try to
> reproduce it with 1.12.0?
>
> --
> Johan
>
Re: svn status and info slowness when multiple files are passed as args
Posted by Johan Corveleyn <jc...@gmail.com>.
On Tue, Jun 4, 2019 at 7:35 AM Thuan Seah Tan <th...@fmod.com> wrote:
>
> Wondering if anyone has experienced slowness when running commandline svn if multiple files are passed to svn status or svn info?
>
> For example:
> svn status directory/file1.txt directory/file2.txt directory/file3.txt
>
> In my tests, the output is generated at about the rate of 1 file per second when doing so. Running svn status on the entire directory recursively turns out to be much faster.
Doesn't ring a bell with me. Can you give some more context?
- Which OS?
- Which filesystem? Is it local or a network filesystem?
- Which version of SVN? If not the latest (1.12.0), can you try to
reproduce it with 1.12.0?
--
Johan