You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by st...@apache.org on 2011/11/01 19:26:55 UTC

svn commit: r1196191 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c

Author: stsp
Date: Tue Nov  1 18:26:54 2011
New Revision: 1196191

URL: http://svn.apache.org/viewvc?rev=1196191&view=rev
Log:
During 'svn rm', sort the delete-list in C, rather than in SQL.
This speeds up 'svn rm dir/*'.

Also fix notification in case of cancellation. We were omitting notification
for some nodes in the delete-list if the user cancelled early. This is wrong
because all nodes in the delete-list are already deleted at this point.

* subversion/libsvn_wc/wc-queries.sql
  (STMT_SELECT_DELETE_LIST): Don't ORDER BY.

* subversion/libsvn_wc/wc_db.c
  (do_delete_notify): Sort the delete-list in C, if it is not already sorted.
   Notify for all paths in the delete-list before allowing cancellation.

Modified:
    subversion/trunk/subversion/libsvn_wc/wc-queries.sql
    subversion/trunk/subversion/libsvn_wc/wc_db.c

Modified: subversion/trunk/subversion/libsvn_wc/wc-queries.sql
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_wc/wc-queries.sql?rev=1196191&r1=1196190&r2=1196191&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_wc/wc-queries.sql (original)
+++ subversion/trunk/subversion/libsvn_wc/wc-queries.sql Tue Nov  1 18:26:54 2011
@@ -1208,7 +1208,6 @@ WHERE wc_id = ?1
 
 -- STMT_SELECT_DELETE_LIST
 SELECT local_relpath FROM delete_list
-ORDER BY local_relpath
 
 -- STMT_FINALIZE_DELETE
 DROP TABLE IF EXISTS delete_list

Modified: subversion/trunk/subversion/libsvn_wc/wc_db.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_wc/wc_db.c?rev=1196191&r1=1196190&r2=1196191&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_wc/wc_db.c (original)
+++ subversion/trunk/subversion/libsvn_wc/wc_db.c Tue Nov  1 18:26:54 2011
@@ -32,6 +32,7 @@
 #include "svn_dirent_uri.h"
 #include "svn_path.h"
 #include "svn_hash.h"
+#include "svn_sorts.h"
 #include "svn_wc.h"
 #include "svn_checksum.h"
 #include "svn_pools.h"
@@ -6559,38 +6560,71 @@ do_delete_notify(void *baton,
   svn_sqlite__stmt_t *stmt;
   svn_boolean_t have_row;
   apr_pool_t *iterpool;
+  apr_array_header_t *delete_list;
+  svn_boolean_t sorted;
+  int i;
 
   SVN_ERR(svn_sqlite__get_statement(&stmt, wcroot->sdb,
                                     STMT_SELECT_DELETE_LIST));
   SVN_ERR(svn_sqlite__step(&have_row, stmt));
 
-  iterpool = svn_pool_create(scratch_pool);
+  /* Build a sorted list of deleted nodes. STMT_SELECT_DELETE_LIST
+   * doesn't sort because sorting in SQL is more expensive than in C. */
+  delete_list = apr_array_make(scratch_pool, 1, sizeof(const char *));
+  sorted = TRUE;
+  i = 0;
   while (have_row)
     {
       const char *notify_relpath;
       const char *notify_abspath;
-
-      svn_pool_clear(iterpool);
-
-      if (cancel_func)
-        SVN_ERR(cancel_func(cancel_baton));
+      const char *prev_abspath;
 
       notify_relpath = svn_sqlite__column_text(stmt, 0, NULL);
       notify_abspath = svn_dirent_join(wcroot->abspath,
                                        notify_relpath,
-                                       iterpool);
+                                       scratch_pool);
+      if (i == 0)
+        {
+          APR_ARRAY_PUSH(delete_list, const char *) = notify_abspath;
+          i++;
+          SVN_ERR(svn_sqlite__step(&have_row, stmt));
+          continue;
+        }
 
+      prev_abspath = APR_ARRAY_IDX(delete_list, i - 1, const char *);
+      if (sorted)
+        sorted = (svn_path_compare_paths(prev_abspath, notify_abspath) <= 0);
+      APR_ARRAY_PUSH(delete_list, const char *) = notify_abspath;
+      i++;
+      SVN_ERR(svn_sqlite__step(&have_row, stmt));
+    }
+
+  SVN_ERR(svn_sqlite__reset(stmt));
+
+  if (!sorted)
+    qsort(delete_list->elts, delete_list->nelts, delete_list->elt_size,
+          svn_sort_compare_paths);
+
+  iterpool = svn_pool_create(scratch_pool);
+  for (i = 0; i < delete_list->nelts; i++)
+    {
+      svn_pool_clear(iterpool);
       notify_func(notify_baton,
-                  svn_wc_create_notify(notify_abspath,
+                  svn_wc_create_notify(APR_ARRAY_IDX(delete_list, i,
+                                                     const char *),
                                        svn_wc_notify_delete,
                                        iterpool),
                   iterpool);
-
-      SVN_ERR(svn_sqlite__step(&have_row, stmt));
     }
   svn_pool_destroy(iterpool);
 
-  return svn_error_trace(svn_sqlite__reset(stmt));
+  /* We only allow cancellation after notification for all deleted nodes
+   * has happened. The nodes are already deleted so we should notify for
+   * all of them. */
+  if (cancel_func)
+    SVN_ERR(cancel_func(cancel_baton));
+
+  return SVN_NO_ERROR;
 }
 
 



Re: svn commit: r1196191 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c

Posted by Hyrum K Wright <hy...@wandisco.com>.
On Tue, Nov 1, 2011 at 1:53 PM, Philip Martin <ph...@wandisco.com>wrote:

> Stefan Sperling <st...@elego.de> writes:
>
> > On Tue, Nov 01, 2011 at 06:36:26PM +0000, Philip Martin wrote:
> >> Is sorting in C better than putting UNIQUE on the local_relpath column?
> >
> > I don't know. I don't mind either way. Let's pick whichever is faster.
>
> I'd expect performance to be similar.  I have a small preference for
> UNIQUE because it leaves sorting and memory allocation to the database.


Same here, assuming the performance is similar.

-Hyrum


-- 

uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com/

Re: svn commit: r1196191 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c

Posted by Philip Martin <ph...@wandisco.com>.
Stefan Sperling <st...@elego.de> writes:

> On Tue, Nov 01, 2011 at 06:36:26PM +0000, Philip Martin wrote:
>> Is sorting in C better than putting UNIQUE on the local_relpath column?
>
> I don't know. I don't mind either way. Let's pick whichever is faster.

I'd expect performance to be similar.  I have a small preference for
UNIQUE because it leaves sorting and memory allocation to the database.

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

Re: svn commit: r1196191 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c

Posted by Stefan Sperling <st...@elego.de>.
On Wed, Nov 02, 2011 at 01:41:29PM +0000, Philip Martin wrote:
> I've been timing the performance on NFS working copies and I can see no
> significant speed difference between using an SQLite index and sorting
> in C.  So I propose to switch to the index as I believe letting the DB
> do the work is better.

I agree.

Re: svn commit: r1196191 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c

Posted by Philip Martin <ph...@wandisco.com>.
Stefan Sperling <st...@elego.de> writes:

> On Wed, Nov 02, 2011 at 12:59:31PM +0000, Philip Martin wrote:
>> Stefan Sperling <st...@elego.de> writes:
>> 
>> > On Tue, Nov 01, 2011 at 06:36:26PM +0000, Philip Martin wrote:
>> >> Is sorting in C better than putting UNIQUE on the local_relpath column?
>> >
>> > I don't know. I don't mind either way. Let's pick whichever is faster.
>> >
>> > The notification/cancellation fix is needed in either case.
>> 
>> I think the cacellation change is wrong.  This is happening inside an
>> SQLite transaction.  When the user cancels a delete operation during
>> notification the cancel callback will return an svn_error_t causing the
>> transaction to abort.  That means that no change will have been made to
>> the database.  I don't see how postponing that abort until all the
>> notifications have been made is an improvement.
>
> The transaction is already done when notification happens.
> See wc_db.c:with_finalization() -- notification is performed by
> the work_cb() callback, not the txn_cb() callback.

OK.

I've been timing the performance on NFS working copies and I can see no
significant speed difference between using an SQLite index and sorting
in C.  So I propose to switch to the index as I believe letting the DB
do the work is better.

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

Re: svn commit: r1196191 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c

Posted by Stefan Sperling <st...@elego.de>.
On Wed, Nov 02, 2011 at 12:59:31PM +0000, Philip Martin wrote:
> Stefan Sperling <st...@elego.de> writes:
> 
> > On Tue, Nov 01, 2011 at 06:36:26PM +0000, Philip Martin wrote:
> >> Is sorting in C better than putting UNIQUE on the local_relpath column?
> >
> > I don't know. I don't mind either way. Let's pick whichever is faster.
> >
> > The notification/cancellation fix is needed in either case.
> 
> I think the cacellation change is wrong.  This is happening inside an
> SQLite transaction.  When the user cancels a delete operation during
> notification the cancel callback will return an svn_error_t causing the
> transaction to abort.  That means that no change will have been made to
> the database.  I don't see how postponing that abort until all the
> notifications have been made is an improvement.

The transaction is already done when notification happens.
See wc_db.c:with_finalization() -- notification is performed by
the work_cb() callback, not the txn_cb() callback.

Re: svn commit: r1196191 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c

Posted by Philip Martin <ph...@wandisco.com>.
Stefan Sperling <st...@elego.de> writes:

> On Tue, Nov 01, 2011 at 06:36:26PM +0000, Philip Martin wrote:
>> Is sorting in C better than putting UNIQUE on the local_relpath column?
>
> I don't know. I don't mind either way. Let's pick whichever is faster.
>
> The notification/cancellation fix is needed in either case.

I think the cacellation change is wrong.  This is happening inside an
SQLite transaction.  When the user cancels a delete operation during
notification the cancel callback will return an svn_error_t causing the
transaction to abort.  That means that no change will have been made to
the database.  I don't see how postponing that abort until all the
notifications have been made is an improvement.

-- 
uberSVN: Apache Subversion Made Easy
http://www.uberSVN.com

Re: svn commit: r1196191 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c

Posted by Stefan Sperling <st...@elego.de>.
On Tue, Nov 01, 2011 at 06:36:26PM +0000, Philip Martin wrote:
> Is sorting in C better than putting UNIQUE on the local_relpath column?

I don't know. I don't mind either way. Let's pick whichever is faster.

The notification/cancellation fix is needed in either case.

Re: svn commit: r1196191 - in /subversion/trunk/subversion/libsvn_wc: wc-queries.sql wc_db.c

Posted by Philip Martin <ph...@wandisco.com>.
stsp@apache.org writes:

> Author: stsp
> Date: Tue Nov  1 18:26:54 2011
> New Revision: 1196191
>
> URL: http://svn.apache.org/viewvc?rev=1196191&view=rev
> Log:
> During 'svn rm', sort the delete-list in C, rather than in SQL.
> This speeds up 'svn rm dir/*'.
>
> Also fix notification in case of cancellation. We were omitting notification
> for some nodes in the delete-list if the user cancelled early. This is wrong
> because all nodes in the delete-list are already deleted at this point.
>
> * subversion/libsvn_wc/wc-queries.sql
>   (STMT_SELECT_DELETE_LIST): Don't ORDER BY.
>
> * subversion/libsvn_wc/wc_db.c
>   (do_delete_notify): Sort the delete-list in C, if it is not already sorted.
>    Notify for all paths in the delete-list before allowing cancellation.

Is sorting in C better than putting UNIQUE on the local_relpath column?

-- 
Philip