You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@subversion.apache.org by mi...@agilent.com on 2011/09/01 17:25:55 UTC
Really lousy performance with svn info --depth infinity
Hi guys,
Quick question. I would expect the following 2 commands to perform equivalent functions.
svn info -depth infinity
svn ls -depth infinity | xargs svn info
This issue is that the first command takes 1 minute to run while the second command takes 6 seconds to run. I am running 1.7.0-rc2.
-Mike
RE: Really lousy performance with svn info --depth infinity
Posted by mi...@agilent.com.
-> sqlite3 .svn/wc.db "select count (*) from nodes"
5242
-> sqlite3 .svn/wc.db "select count (*) from nodes where op_depth > 0"
0
-> sqlite3 .svn/wc.db "select count (*) from actual_node"
0
I ran the following command
for i in $(find . -type d | grep -v .svn); do ls -1 $i | wc -l; done | sort -n
The directory with the largest number of sub-stuff had 211 entries.
-----Original Message-----
From: Philip Martin [mailto:philip.martin@wandisco.com]
Sent: Thursday, September 01, 2011 11:13 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: d.s@daniel.shahaf.name; dev@subversion.apache.org
Subject: Re: Really lousy performance with svn info --depth infinity
<mi...@agilent.com> writes:
> And here is the final comparison using an nfs mounted working copy.
> This is where the difference gets really bad.
>
> 1.6.17
>
> -> time /file_access/subversion/1.6.17/bin/svn info --depth infinity >
> -> /dev/null
>
> real 0m2.548s
> user 0m0.350s
> sys 0m0.142s
>
> 1.7.0-rc2
>
> -> time svn info --depth infinity > /dev/null
>
> real 6m51.036s
> user 0m13.947s
> sys 0m10.880s
I see the opposite on an NFS disk: the single recursive call is 20s and the multiple non-recursive calls are 33s, so the single call is faster as expected. It's still 20x slower than a local disk but that will be because info is still using per-node sqlite transactions.
What do these command show:
$ sqlite3 .svn/wc.db "select count (*) from nodes"
$ sqlite3 .svn/wc.db "select count (*) from nodes where op_depth > 0"
$ sqlite3 .svn/wc.db "select count (*) from actual_node"
Does your working copy have "large" directories, i.e. a directory with a large number of immediate subdirs/files? (It should be possible to forumulate an SQL statement that tells me, but my SQL isn't good enough).
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: [PATCH] Improve speed of recursive info
Posted by Philip Martin <ph...@wandisco.com>.
<mi...@agilent.com> writes:
> That definitely improved the performance. The time it takes to run
> went from 6+ minutes to 1m7s. It's pretty much the same speed as the
> individual queries now. The performance is still significantly worse
> than 1.6.17 was and I don't know if that should be addressed.
Thanks for testing. The performance relative to 1.6 will need a much
more invasive fix, we need to rework the code to avoid doing per-node
queries for each file/directory. We had the same problem with status.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: [PATCH] Improve speed of recursive info
Posted by mi...@agilent.com.
That definitely improved the performance. The time it takes to run went from 6+ minutes to 1m7s. It's pretty much the same speed as the individual queries now. The performance is still significantly worse than 1.6.17 was and I don't know if that should be addressed.
-----Original Message-----
From: Philip Martin [mailto:philip.martin@wandisco.com]
Sent: Friday, September 02, 2011 4:38 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: d.s@daniel.shahaf.name; dev@subversion.apache.org
Subject: Re: [PATCH] Improve speed of recursive info
Philip Martin <ph...@wandisco.com> writes:
> Philip Martin <ph...@wandisco.com> writes:
>
>> So that indicates that we
>> have a scaling problem, and I suspect the ORDER/GROUP part of
>> STMT_SELECT_NODE_CHILDREN_WALKER_INFO. I don't know whether that is
>> enough to explain the extreme runtime you are getting.
>
> Michael, are you building 1.7 from source? Will you try a patch? The
> following patch against 1.7 addresses the above problem and improves
> the speed of recursive info on my machine:
It looks like we might be able to fix this using another SQL index rather than changing the C code. You can create the index in an existing working copy using:
sqlite3 .svn/wc.db "create index i_p2 on nodes(wc_id, parent_relpath, local_relpath, op_depth)"
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: [PATCH] Improve speed of recursive info
Posted by Philip Martin <ph...@wandisco.com>.
Philip Martin <ph...@wandisco.com> writes:
> Philip Martin <ph...@wandisco.com> writes:
>
>> So that indicates that we
>> have a scaling problem, and I suspect the ORDER/GROUP part of
>> STMT_SELECT_NODE_CHILDREN_WALKER_INFO. I don't know whether that is
>> enough to explain the extreme runtime you are getting.
>
> Michael, are you building 1.7 from source? Will you try a patch? The
> following patch against 1.7 addresses the above problem and improves the
> speed of recursive info on my machine:
It looks like we might be able to fix this using another SQL index
rather than changing the C code. You can create the index in an
existing working copy using:
sqlite3 .svn/wc.db "create index i_p2 on nodes(wc_id, parent_relpath, local_relpath, op_depth)"
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
[PATCH] Improve speed of recursive info
Posted by Philip Martin <ph...@wandisco.com>.
Philip Martin <ph...@wandisco.com> writes:
> So that indicates that we
> have a scaling problem, and I suspect the ORDER/GROUP part of
> STMT_SELECT_NODE_CHILDREN_WALKER_INFO. I don't know whether that is
> enough to explain the extreme runtime you are getting.
Michael, are you building 1.7 from source? Will you try a patch? The
following patch against 1.7 addresses the above problem and improves the
speed of recursive info on my machine:
Index: subversion/libsvn_wc/wc-queries.sql
===================================================================
--- subversion/libsvn_wc/wc-queries.sql (revision 1164384)
+++ subversion/libsvn_wc/wc-queries.sql (working copy)
@@ -126,13 +126,10 @@
WHERE wc_id = ?1 AND parent_relpath = ?2
-- STMT_SELECT_NODE_CHILDREN_WALKER_INFO
-/* ### See comment at STMT_SELECT_NODE_CHILDREN_INFO.
- ### Should C code handle GROUP BY local_relpath ORDER BY op_depths DESC? */
+/* See comment at STMT_SELECT_NODE_CHILDREN_INFO about GROUP/ORDER */
SELECT local_relpath, op_depth, presence, kind
FROM nodes
WHERE wc_id = ?1 AND parent_relpath = ?2
-GROUP BY local_relpath
-ORDER BY op_depth DESC
-- STMT_SELECT_ACTUAL_CHILDREN_INFO
SELECT prop_reject, changelist, conflict_old, conflict_new,
Index: subversion/libsvn_wc/wc_db.c
===================================================================
--- subversion/libsvn_wc/wc_db.c (revision 1164384)
+++ subversion/libsvn_wc/wc_db.c (working copy)
@@ -7156,6 +7156,11 @@
svn_sqlite__reset(stmt)));
}
+struct read_children_walker_info_item_t {
+ struct svn_wc__db_walker_info_t child;
+ apr_int64_t op_depth;
+};
+
svn_error_t *
svn_wc__db_read_children_walker_info(apr_hash_t **nodes,
svn_wc__db_t *db,
@@ -7167,7 +7172,6 @@
const char *dir_relpath;
svn_sqlite__stmt_t *stmt;
svn_boolean_t have_row;
- apr_int64_t op_depth;
SVN_ERR_ASSERT(svn_dirent_is_absolute(dir_abspath));
@@ -7184,26 +7188,38 @@
*nodes = apr_hash_make(result_pool);
while (have_row)
{
- struct svn_wc__db_walker_info_t *child;
+ struct read_children_walker_info_item_t *child_item;
const char *child_relpath = svn_sqlite__column_text(stmt, 0, NULL);
const char *name = svn_relpath_basename(child_relpath, NULL);
+ apr_int64_t op_depth;
+ svn_boolean_t new_item;
svn_error_t *err;
- child = apr_hash_get(*nodes, name, APR_HASH_KEY_STRING);
- if (child == NULL)
- child = apr_palloc(result_pool, sizeof(*child));
+ child_item = apr_hash_get(*nodes, name, APR_HASH_KEY_STRING);
+ if (child_item)
+ new_item = FALSE;
+ else
+ {
+ child_item = apr_palloc(result_pool, sizeof(*child_item));
+ apr_hash_set(*nodes, apr_pstrdup(result_pool, name),
+ APR_HASH_KEY_STRING, child_item);
+ new_item = TRUE;
+ }
op_depth = svn_sqlite__column_int(stmt, 1);
- child->status = svn_sqlite__column_token(stmt, 2, presence_map);
- if (op_depth > 0)
+ if (new_item || op_depth > child_item->op_depth)
{
- err = convert_to_working_status(&child->status, child->status);
- if (err)
- SVN_ERR(svn_error_compose_create(err, svn_sqlite__reset(stmt)));
+ struct svn_wc__db_walker_info_t *child = &child_item->child;
+ child_item->op_depth = op_depth;
+ child->status = svn_sqlite__column_token(stmt, 2, presence_map);
+ if (op_depth > 0)
+ {
+ err = convert_to_working_status(&child->status, child->status);
+ if (err)
+ SVN_ERR(svn_error_compose_create(err, svn_sqlite__reset(stmt)));
+ }
+ child->kind = svn_sqlite__column_token(stmt, 3, kind_map);
}
- child->kind = svn_sqlite__column_token(stmt, 3, kind_map);
- apr_hash_set(*nodes, apr_pstrdup(result_pool, name),
- APR_HASH_KEY_STRING, child);
err = svn_sqlite__step(&have_row, stmt);
if (err)
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Really lousy performance with svn info --depth infinity
Posted by Philip Martin <ph...@wandisco.com>.
<mi...@agilent.com> writes:
> -> sqlite3 .svn/wc.db "select parent_relpath, count(*) AS n from nodes group by parent_relpath order by n desc limit 1"
> ip/DESIGNWARE/verilog|211
That's about 2.5 times the size of a Subversion trunk wc, both in number
of nodes and biggest directory.
If I "svn add" a few hundred files in one directory of my working copy
increasing the total number of nodes by about 33% the runtime of the
recursive info call increase by about 70%. So that indicates that we
have a scaling problem, and I suspect the ORDER/GROUP part of
STMT_SELECT_NODE_CHILDREN_WALKER_INFO. I don't know whether that is
enough to explain the extreme runtime you are getting.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: Really lousy performance with svn info --depth infinity
Posted by mi...@agilent.com.
-> sqlite3 .svn/wc.db "select parent_relpath, count(*) AS n from nodes group by parent_relpath order by n desc limit 1"
ip/DESIGNWARE/verilog|211
-----Original Message-----
From: Philip Martin [mailto:philip.martin@wandisco.com]
Sent: Thursday, September 01, 2011 11:23 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: d.s@daniel.shahaf.name; dev@subversion.apache.org
Subject: Re: Really lousy performance with svn info --depth infinity
Philip Martin <ph...@wandisco.com> writes:
> Does your working copy have "large" directories, i.e. a directory with
> a large number of immediate subdirs/files? (It should be possible to
> forumulate an SQL statement that tells me, but my SQL isn't good
> enough).
Thanks to Daniel:
$ sqlite3 .svn/wc.db "select parent_relpath, count(*) AS n from nodes group by parent_relpath order by n desc limit 1"
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Really lousy performance with svn info --depth infinity
Posted by Philip Martin <ph...@wandisco.com>.
Philip Martin <ph...@wandisco.com> writes:
> Does your working copy have "large" directories, i.e. a directory with a
> large number of immediate subdirs/files? (It should be possible to
> forumulate an SQL statement that tells me, but my SQL isn't good
> enough).
Thanks to Daniel:
$ sqlite3 .svn/wc.db "select parent_relpath, count(*) AS n from nodes group by parent_relpath order by n desc limit 1"
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Really lousy performance with svn info --depth infinity
Posted by Philip Martin <ph...@wandisco.com>.
<mi...@agilent.com> writes:
> And here is the final comparison using an nfs mounted working copy.
> This is where the difference gets really bad.
>
> 1.6.17
>
> -> time /file_access/subversion/1.6.17/bin/svn info --depth infinity > /dev/null
>
> real 0m2.548s
> user 0m0.350s
> sys 0m0.142s
>
> 1.7.0-rc2
>
> -> time svn info --depth infinity > /dev/null
>
> real 6m51.036s
> user 0m13.947s
> sys 0m10.880s
I see the opposite on an NFS disk: the single recursive call is 20s and
the multiple non-recursive calls are 33s, so the single call is faster
as expected. It's still 20x slower than a local disk but that will be
because info is still using per-node sqlite transactions.
What do these command show:
$ sqlite3 .svn/wc.db "select count (*) from nodes"
$ sqlite3 .svn/wc.db "select count (*) from nodes where op_depth > 0"
$ sqlite3 .svn/wc.db "select count (*) from actual_node"
Does your working copy have "large" directories, i.e. a directory with a
large number of immediate subdirs/files? (It should be possible to
forumulate an SQL statement that tells me, but my SQL isn't good
enough).
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: Really lousy performance with svn info --depth infinity
Posted by mi...@agilent.com.
And here is the final comparison using an nfs mounted working copy. This is where the difference gets really bad.
1.6.17
-> time /file_access/subversion/1.6.17/bin/svn info --depth infinity > /dev/null
real 0m2.548s
user 0m0.350s
sys 0m0.142s
1.7.0-rc2
-> time svn info --depth infinity > /dev/null
real 6m51.036s
user 0m13.947s
sys 0m10.880s
-Mike
-----Original Message-----
From: Philip Martin [mailto:philip.martin@wandisco.com]
Sent: Thursday, September 01, 2011 10:26 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: d.s@daniel.shahaf.name; dev@subversion.apache.org
Subject: Re: Really lousy performance with svn info --depth infinity
<mi...@agilent.com> writes:
> And here is the comparsion of 1.6.17 vs 1.7.0-rc2. This test was run
> on a locally mounted drive, it gets significantly worse on an nfs
> mounted drive.
>
> 1.6.17
>
> -> time /file_access/subversion/1.6.17/bin/svn info --depth infinity >
> -> /dev/null
>
> real 0m0.225s
> user 0m0.204s
> sys 0m0.016s
>
> 1.7.0-rc2
>
> -> time svn info --depth infinity > /dev/null
>
> real 0m7.741s
> user 0m6.571s
> sys 0m1.155s
The recursive "svn info" still does multiple sqlite transactions per-node, that probably explains why it is slower than 1.6. It's doesn't really explain why the single recursive call is slower than multiple non-recursive calls, since each non-recursive call also makes several sqlite transactions.
At first glance the difference between recursive and non-recursive is node.c:walker_helper. Is STMT_SELECT_NODE_CHILDREN_WALKER_INFO missing an index?
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Really lousy performance with svn info --depth infinity
Posted by Philip Martin <ph...@wandisco.com>.
Philip Martin <ph...@wandisco.com> writes:
> At first glance the difference between recursive and non-recursive is
> node.c:walker_helper. Is STMT_SELECT_NODE_CHILDREN_WALKER_INFO missing
> an index?
Look at the comment:
-- STMT_SELECT_NODE_CHILDREN_WALKER_INFO
/* ### See comment at STMT_SELECT_NODE_CHILDREN_INFO.
### Should C code handle GROUP BY local_relpath ORDER BY op_depths DESC? */
SELECT local_relpath, op_depth, presence, kind
FROM nodes
WHERE wc_id = ?1 AND parent_relpath = ?2
GROUP BY local_relpath
ORDER BY op_depth DESC
Perhaps GROUP BY and ORDER BY are the problem. Perhaps they defeat the
index? Perhaps we need to do it like STMT_SELECT_NODE_CHILDREN_INFO, in
C.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Really lousy performance with svn info --depth infinity
Posted by Philip Martin <ph...@wandisco.com>.
<mi...@agilent.com> writes:
> And here is the comparsion of 1.6.17 vs 1.7.0-rc2. This test was run
> on a locally mounted drive, it gets significantly worse on an nfs
> mounted drive.
>
> 1.6.17
>
> -> time /file_access/subversion/1.6.17/bin/svn info --depth infinity > /dev/null
>
> real 0m0.225s
> user 0m0.204s
> sys 0m0.016s
>
> 1.7.0-rc2
>
> -> time svn info --depth infinity > /dev/null
>
> real 0m7.741s
> user 0m6.571s
> sys 0m1.155s
The recursive "svn info" still does multiple sqlite transactions
per-node, that probably explains why it is slower than 1.6. It's
doesn't really explain why the single recursive call is slower than
multiple non-recursive calls, since each non-recursive call also makes
several sqlite transactions.
At first glance the difference between recursive and non-recursive is
node.c:walker_helper. Is STMT_SELECT_NODE_CHILDREN_WALKER_INFO missing
an index?
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: Really lousy performance with svn info --depth infinity
Posted by mi...@agilent.com.
And here is the comparsion of 1.6.17 vs 1.7.0-rc2. This test was run on a locally mounted drive, it gets significantly worse on an nfs mounted drive.
1.6.17
-> time /file_access/subversion/1.6.17/bin/svn info --depth infinity > /dev/null
real 0m0.225s
user 0m0.204s
sys 0m0.016s
1.7.0-rc2
-> time svn info --depth infinity > /dev/null
real 0m7.741s
user 0m6.571s
sys 0m1.155s
-----Original Message-----
From: Philip Martin [mailto:philip.martin@wandisco.com]
Sent: Thursday, September 01, 2011 10:09 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: d.s@daniel.shahaf.name; dev@subversion.apache.org
Subject: Re: Really lousy performance with svn info --depth infinity
<mi...@agilent.com> writes:
> Svn info --depth inifinty
WC on local disk or network disk?
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Really lousy performance with svn info --depth infinity
Posted by Philip Martin <ph...@wandisco.com>.
<mi...@agilent.com> writes:
> Svn info --depth inifinty
WC on local disk or network disk?
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: Really lousy performance with svn info --depth infinity
Posted by mi...@agilent.com.
Svn info --depth inifinty
-----Original Message-----
From: Daniel Shahaf [mailto:d.s@daniel.shahaf.name]
Sent: Thursday, September 01, 2011 10:05 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: philip.martin@wandisco.com; dev@subversion.apache.org
Subject: Re: Really lousy performance with svn info --depth infinity
What command is slower in 1.7 compared to 1.6?
michael_rytting@agilent.com wrote on Thu, Sep 01, 2011 at 09:55:30 -0600:
> We are still using 1.6.x for our repository. I wonder if there is a funny interaction going on here. I might need to do some more digging, but the performance we are seeing is currently 1000x slower with 1.7 working copy vs 1.6 working copy.
>
> -----Original Message-----
> From: Philip Martin [mailto:philip.martin@wandisco.com]
> Sent: Thursday, September 01, 2011 9:52 AM
> To: RYTTING,MICHAEL (A-ColSprings,ex1)
> Cc: dev@subversion.apache.org
> Subject: Re: Really lousy performance with svn info --depth infinity
>
> <mi...@agilent.com> writes:
>
> > Quick question. I would expect the following 2 commands to perform
> > equivalent functions.
> >
> > svn info -depth infinity
> > svn ls -depth infinity | xargs svn info
> >
> > This issue is that the first command takes 1 minute to run while the
> > second command takes 6 seconds to run. I am running 1.7.0-rc2.
>
> I'm using a working copy that is Subversion trunk and a local repository as my dataset on my Linux machine. Both commands take about 1 second with a hot cache.
>
> --
> uberSVN: Apache Subversion Made Easy
> http://www.uberSVN.com
Re: Really lousy performance with svn info --depth infinity
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
What command is slower in 1.7 compared to 1.6?
michael_rytting@agilent.com wrote on Thu, Sep 01, 2011 at 09:55:30 -0600:
> We are still using 1.6.x for our repository. I wonder if there is a funny interaction going on here. I might need to do some more digging, but the performance we are seeing is currently 1000x slower with 1.7 working copy vs 1.6 working copy.
>
> -----Original Message-----
> From: Philip Martin [mailto:philip.martin@wandisco.com]
> Sent: Thursday, September 01, 2011 9:52 AM
> To: RYTTING,MICHAEL (A-ColSprings,ex1)
> Cc: dev@subversion.apache.org
> Subject: Re: Really lousy performance with svn info --depth infinity
>
> <mi...@agilent.com> writes:
>
> > Quick question. I would expect the following 2 commands to perform
> > equivalent functions.
> >
> > svn info -depth infinity
> > svn ls -depth infinity | xargs svn info
> >
> > This issue is that the first command takes 1 minute to run while the
> > second command takes 6 seconds to run. I am running 1.7.0-rc2.
>
> I'm using a working copy that is Subversion trunk and a local repository as my dataset on my Linux machine. Both commands take about 1 second with a hot cache.
>
> --
> uberSVN: Apache Subversion Made Easy
> http://www.uberSVN.com
RE: Really lousy performance with svn info --depth infinity
Posted by mi...@agilent.com.
We are still using 1.6.x for our repository. I wonder if there is a funny interaction going on here. I might need to do some more digging, but the performance we are seeing is currently 1000x slower with 1.7 working copy vs 1.6 working copy.
-----Original Message-----
From: Philip Martin [mailto:philip.martin@wandisco.com]
Sent: Thursday, September 01, 2011 9:52 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1)
Cc: dev@subversion.apache.org
Subject: Re: Really lousy performance with svn info --depth infinity
<mi...@agilent.com> writes:
> Quick question. I would expect the following 2 commands to perform
> equivalent functions.
>
> svn info -depth infinity
> svn ls -depth infinity | xargs svn info
>
> This issue is that the first command takes 1 minute to run while the
> second command takes 6 seconds to run. I am running 1.7.0-rc2.
I'm using a working copy that is Subversion trunk and a local repository as my dataset on my Linux machine. Both commands take about 1 second with a hot cache.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Really lousy performance with svn info --depth infinity
Posted by Philip Martin <ph...@wandisco.com>.
<mi...@agilent.com> writes:
> Quick question. I would expect the following 2 commands to perform
> equivalent functions.
>
> svn info -depth infinity
> svn ls -depth infinity | xargs svn info
>
> This issue is that the first command takes 1 minute to run while the
> second command takes 6 seconds to run. I am running 1.7.0-rc2.
I'm using a working copy that is Subversion trunk and a local repository
as my dataset on my Linux machine. Both commands take about 1 second
with a hot cache.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: Really lousy performance with svn info --depth infinity
Posted by mi...@agilent.com.
Very well, we'll move it to users. I just figured it was a bug in 1.7 since the performance of the svn info -depth infinity was SIGNIFICANTLY faster in 1.6.x. Currently it takes 1 minute with 1.7 and less than a second with 1.6.x.
-Mike
From: Bert Huijben [mailto:bert@qqmail.nl]
Sent: Thursday, September 01, 2011 9:43 AM
To: RYTTING,MICHAEL (A-ColSprings,ex1); dev@subversion.apache.org
Subject: RE: Really lousy performance with svn info --depth infinity
These operations do very different things. The first reads your working copy to obtain a lot of details (unless you pass a revision; in that case it looks at the repository)
The second asks the repository what files there are below the current directory in the repository (using the revision of the current directory).
In my usage (repository hosted via http:// on the other side of the world) the second is much slower... But if you have a local repository the opposite might be true.
I think this question belongs on users{_AT_}subversion.apache.org though, as it is about using Subversion and not on Subversion development.
Bert
From: michael_rytting@agilent.com [mailto:michael_rytting@agilent.com]
Sent: donderdag 1 september 2011 17:26
To: dev@subversion.apache.org
Subject: Really lousy performance with svn info --depth infinity
Hi guys,
Quick question. I would expect the following 2 commands to perform equivalent functions.
svn info -depth infinity
svn ls -depth infinity | xargs svn info
This issue is that the first command takes 1 minute to run while the second command takes 6 seconds to run. I am running 1.7.0-rc2.
-Mike
RE: Really lousy performance with svn info --depth infinity
Posted by mi...@agilent.com.
Hmm,
Seems to be worse when my working copy is on an nfs mounted drive, if my working copy is on a locally mounted drive than the performance of the 2 commands is not as different.
-> time svn info --depth infinity > /dev/null
real 0m7.697s
user 0m6.606s
sys 0m1.085s
-> time svn ls --depth infinity | xargs svn info
real 0m1.540s
user 0m0.514s
sys 0m0.223s
My wc.db is pretty big weighing in at 3.1M
-----Original Message-----
From: Philip Martin [mailto:philip.martin@wandisco.com]
Sent: Thursday, September 01, 2011 10:01 AM
To: Bert Huijben
Cc: RYTTING,MICHAEL (A-ColSprings,ex1); dev@subversion.apache.org
Subject: Re: Really lousy performance with svn info --depth infinity
"Bert Huijben" <be...@qqmail.nl> writes:
> These operations do very different things. The first reads your
> working copy to obtain a lot of details (unless you pass a revision;
> in that case it looks at the repository)
>
> The second asks the repository what files there are below the current
> directory in the repository (using the revision of the current
> directory).
Both commands access the working copy to read the info for every file and directory.
> In my usage (repository hosted via http:// on the other side of the
> world) the second is much slower. But if you have a local repository
> the opposite might be true.
>
> I think this question belongs on users{_AT_}subversion.apache.org
> though, as it is about using Subversion and not on Subversion
> development.
I disagree. Since both commands pull all the info from the working copy it is a problem if the single recursive call is slower than multiple
non-recursive calls. If this is reproducible then we want to know why,
however I cannot reproduce it on my machine.
> From: michael_rytting@agilent.com [mailto:michael_rytting@agilent.com]
> Sent: donderdag 1 september 2011 17:26 To: dev@subversion.apache.org
> Subject: Really lousy performance with svn info --depth infinity
>
> Quick question. I would expect the following 2 commands to perform
> equivalent functions.
>
> svn info -depth infinity
>
> svn ls -depth infinity | xargs svn info
>
> This issue is that the first command takes 1 minute to run while the
> second command takes 6 seconds to run. I am running 1.7.0-rc2.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
Re: Really lousy performance with svn info --depth infinity
Posted by Philip Martin <ph...@wandisco.com>.
"Bert Huijben" <be...@qqmail.nl> writes:
> These operations do very different things. The first reads your
> working copy to obtain a lot of details (unless you pass a revision;
> in that case it looks at the repository)
>
> The second asks the repository what files there are below the current
> directory in the repository (using the revision of the current
> directory).
Both commands access the working copy to read the info for every file
and directory.
> In my usage (repository hosted via http:// on the other side of the
> world) the second is much slower. But if you have a local repository
> the opposite might be true.
>
> I think this question belongs on users{_AT_}subversion.apache.org
> though, as it is about using Subversion and not on Subversion
> development.
I disagree. Since both commands pull all the info from the working copy
it is a problem if the single recursive call is slower than multiple
non-recursive calls. If this is reproducible then we want to know why,
however I cannot reproduce it on my machine.
> From: michael_rytting@agilent.com [mailto:michael_rytting@agilent.com]
> Sent: donderdag 1 september 2011 17:26 To: dev@subversion.apache.org
> Subject: Really lousy performance with svn info --depth infinity
>
> Quick question. I would expect the following 2 commands to perform
> equivalent functions.
>
> svn info -depth infinity
>
> svn ls -depth infinity | xargs svn info
>
> This issue is that the first command takes 1 minute to run while the
> second command takes 6 seconds to run. I am running 1.7.0-rc2.
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com
RE: Really lousy performance with svn info --depth infinity
Posted by Bert Huijben <be...@qqmail.nl>.
These operations do very different things. The first reads your working copy
to obtain a lot of details (unless you pass a revision; in that case it
looks at the repository)
The second asks the repository what files there are below the current
directory in the repository (using the revision of the current directory).
In my usage (repository hosted via http:// on the other side of the world)
the second is much slower. But if you have a local repository the opposite
might be true.
I think this question belongs on users{_AT_}subversion.apache.org though, as
it is about using Subversion and not on Subversion development.
Bert
From: michael_rytting@agilent.com [mailto:michael_rytting@agilent.com]
Sent: donderdag 1 september 2011 17:26
To: dev@subversion.apache.org
Subject: Really lousy performance with svn info --depth infinity
Hi guys,
Quick question. I would expect the following 2 commands to perform
equivalent functions.
svn info -depth infinity
svn ls -depth infinity | xargs svn info
This issue is that the first command takes 1 minute to run while the second
command takes 6 seconds to run. I am running 1.7.0-rc2.
-Mike
Re: Really lousy performance with svn info --depth infinity
Posted by Hyrum K Wright <hy...@wandisco.com>.
By the way, although I haven't been that involved in this discussion,
I'd just like to point out that this is a textbook example of why we
ask people to test the prereleases and release candidates. I wish
more people would.
Thanks for the good discussion and followup; this will probably be
fixed in 1.7.0.
-Hyrum
On Thu, Sep 1, 2011 at 10:25 AM, <mi...@agilent.com> wrote:
> Hi guys,
>
>
>
> Quick question. I would expect the following 2 commands to perform
> equivalent functions.
>
>
>
> svn info –depth infinity
>
> svn ls –depth infinity | xargs svn info
>
>
>
> This issue is that the first command takes 1 minute to run while the second
> command takes 6 seconds to run. I am running 1.7.0-rc2.
>
>
>
> -Mike
--
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com/