You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by st...@apache.org on 2016/10/15 15:58:22 UTC
svn commit: r1765088 - in /subversion/trunk/subversion: include/svn_repos.h
libsvn_repos/list.c libsvn_repos/repos.c tests/libsvn_repos/repos-test.c
Author: stefan2
Date: Sat Oct 15 15:58:22 2016
New Revision: 1765088
URL: http://svn.apache.org/viewvc?rev=1765088&view=rev
Log:
Introduce a "directory list" operation at the repos layer. This is the
first step towards an efficient implementation of 'svn list -r'.
The function is straight-forward with support for operation depth and
authz. One important use-case is "find files". To support this with
minimal system load, the function allows to specify a glob pattern for
the file name and has an option to only report path and node type.
* subversion/include/svn_repos.h
(svn_repos_dirent_receiver_t,
svn_repos_list): The new API.
* subversion/libsvn_repos/list.c
(*): New file implementing the new API. Noteworthy:
(fill_dirent): Factored out from ...
(svn_repos_stat): ... this, which has been moved here from repos.c
* subversion/libsvn_repos/repos.c
(svn_repos_stat): Move from here to list.c due to shared code.
* subversion/tests/libsvn_repos/repos-test.c
(test_list): Simple test-case for the new function.
(test_funcs): Register new test.
Added:
subversion/trunk/subversion/libsvn_repos/list.c (with props)
Modified:
subversion/trunk/subversion/include/svn_repos.h
subversion/trunk/subversion/libsvn_repos/repos.c
subversion/trunk/subversion/tests/libsvn_repos/repos-test.c
Modified: subversion/trunk/subversion/include/svn_repos.h
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/include/svn_repos.h?rev=1765088&r1=1765087&r2=1765088&view=diff
==============================================================================
--- subversion/trunk/subversion/include/svn_repos.h (original)
+++ subversion/trunk/subversion/include/svn_repos.h Sat Oct 15 15:58:22 2016
@@ -1707,6 +1707,66 @@ svn_repos_stat(svn_dirent_t **dirent,
const char *path,
apr_pool_t *pool);
+/**
+ * Callback type to be used with @c svn_repos_list. It will be invoked for
+ * every directory entry found.
+ *
+ * The full path of the entry is given in @a path and @a dirent contains
+ * various additional information. If @c svn_repos_list has been called
+ * with @a path_info_only set, only the @a kind element of this struct
+ * will be valid.
+ *
+ * @a baton is the user-provided receiver baton. @a pool may be used for
+ * temporary allocations.
+ *
+ * @since New in 1.10.
+ */
+typedef svn_error_t *(* svn_repos_dirent_receiver_t)(const char *path,
+ svn_dirent_t *dirent,
+ void *baton,
+ apr_pool_t *pool);
+
+/**
+ * Efficiently list everything within a sub-tree. Specify a glob pattern
+ * to search for specific files and folders.
+ *
+ * Walk the sub-tree starting at @a path under @a root up to the given
+ * @a depth. For each directory entry found, @a receiver will be called
+ * with @a receiver_baton. The starting @a path will be reported as well.
+ * Because retrieving all elements of a @c svn_dirent_t can be expensive,
+ * you may set @a path_info_only to receive only the path name and the node
+ * kind.
+ *
+ * If @a pattern is not @c NULL, only those entries will be reported whose
+ * last path segment matches @a pattern. This feature uses @c apr_fnmatch
+ * for glob matching and requiring '.' to matched by dots in the path.
+ *
+ * If @a authz_read_func is not @c NULL, this function will neither report
+ * entries nor recurse into directories that the user has no access to.
+ *
+ * Cancellation support is provided in the usual way through the optional
+ * @a cancel_func and @a cancel_baton.
+ *
+ * @a path must point to a directory and @a depth must be at least
+ * @c svn_depth_empty.
+ *
+ * Use @a pool for temporary memory allocation.
+ *
+ * @since New in 1.10.
+ */
+svn_error_t *
+svn_repos_list(svn_fs_root_t *root,
+ const char *path,
+ const char *pattern,
+ svn_depth_t depth,
+ svn_boolean_t path_info_only,
+ svn_repos_authz_func_t authz_read_func,
+ void *authz_read_baton,
+ svn_repos_dirent_receiver_t receiver,
+ void *receiver_baton,
+ svn_cancel_func_t cancel_func,
+ void *cancel_baton,
+ apr_pool_t *pool);
/**
* Given @a path which exists at revision @a start in @a fs, set
Added: subversion/trunk/subversion/libsvn_repos/list.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_repos/list.c?rev=1765088&view=auto
==============================================================================
--- subversion/trunk/subversion/libsvn_repos/list.c (added)
+++ subversion/trunk/subversion/libsvn_repos/list.c Sat Oct 15 15:58:22 2016
@@ -0,0 +1,248 @@
+/* list.c : listing repository contents
+ *
+ * ====================================================================
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ * ====================================================================
+ */
+
+#include <apr_pools.h>
+#include <apr_fnmatch.h>
+
+#include "svn_pools.h"
+#include "svn_error.h"
+#include "svn_dirent_uri.h"
+#include "svn_time.h"
+
+#include "private/svn_repos_private.h"
+#include "svn_private_config.h" /* for SVN_TEMPLATE_ROOT_DIR */
+
+#include "repos.h"
+
+
+
+/* Utility function. Given DIRENT->KIND, set all other elements of *DIRENT
+ * with the values retrieved for PATH under ROOT. Allocate them in POOL.
+ */
+static svn_error_t *
+fill_dirent(svn_dirent_t *dirent,
+ svn_fs_root_t *root,
+ const char *path,
+ apr_pool_t *pool)
+{
+ const char *datestring;
+
+ if (dirent->kind == svn_node_file)
+ SVN_ERR(svn_fs_file_length(&(dirent->size), root, path, pool));
+
+ SVN_ERR(svn_fs_node_has_props(&dirent->has_props, root, path, pool));
+
+ SVN_ERR(svn_repos_get_committed_info(&(dirent->created_rev),
+ &datestring,
+ &(dirent->last_author),
+ root, path, pool));
+ if (datestring)
+ SVN_ERR(svn_time_from_cstring(&(dirent->time), datestring, pool));
+
+ return SVN_NO_ERROR;
+}
+
+svn_error_t *
+svn_repos_stat(svn_dirent_t **dirent,
+ svn_fs_root_t *root,
+ const char *path,
+ apr_pool_t *pool)
+{
+ svn_node_kind_t kind;
+ svn_dirent_t *ent;
+
+ SVN_ERR(svn_fs_check_path(&kind, root, path, pool));
+
+ if (kind == svn_node_none)
+ {
+ *dirent = NULL;
+ return SVN_NO_ERROR;
+ }
+
+ ent = svn_dirent_create(pool);
+ ent->kind = kind;
+
+ SVN_ERR(fill_dirent(ent, root, path, pool));
+
+ *dirent = ent;
+ return SVN_NO_ERROR;
+}
+
+/* Utility to prevent code duplication.
+ *
+ * If DIRNAME matches the optional PATTERN, construct a svn_dirent_t for
+ * PATH of type KIND under ROOT and, if PATH_INFO_ONLY is not set, fill it.
+ * Call RECEIVER with the result and RECEIVER_BATON.
+ *
+ * Use POOL for temporary allocations.
+ */
+static svn_error_t *
+report_dirent(svn_fs_root_t *root,
+ const char *path,
+ svn_node_kind_t kind,
+ const char *dirname,
+ const char *pattern,
+ svn_boolean_t path_info_only,
+ svn_repos_dirent_receiver_t receiver,
+ void *receiver_baton,
+ apr_pool_t *pool)
+{
+ svn_dirent_t dirent = { 0 };
+
+ if ( pattern
+ && (apr_fnmatch(pattern, dirname, APR_FNM_PERIOD) != APR_SUCCESS))
+ return SVN_NO_ERROR;
+
+ dirent.kind = kind;
+ if (!path_info_only)
+ SVN_ERR(fill_dirent(&dirent, root, path, pool));
+
+ SVN_ERR(receiver(path, &dirent, receiver_baton, pool));
+
+ return SVN_NO_ERROR;
+}
+
+/* Core of svn_repos_list with the same parameter list.
+ *
+ * However, DEPTH is not svn_depth_empty and PATH has already been reported.
+ * Therefore, we can call this recursively.
+ */
+static svn_error_t *
+do_list(svn_fs_root_t *root,
+ const char *path,
+ const char *pattern,
+ svn_depth_t depth,
+ svn_boolean_t path_info_only,
+ svn_repos_authz_func_t authz_read_func,
+ void *authz_read_baton,
+ svn_repos_dirent_receiver_t receiver,
+ void *receiver_baton,
+ svn_cancel_func_t cancel_func,
+ void *cancel_baton,
+ apr_pool_t *pool)
+{
+ apr_hash_t *entries;
+ apr_pool_t *iterpool = svn_pool_create(pool);
+ apr_hash_index_t *hi;
+
+ /* Iterate over all directory entries, filter and report them.
+ * Recurse into sub-directories if requested. */
+ SVN_ERR(svn_fs_dir_entries(&entries, root, path, pool));
+ for (hi = apr_hash_first(pool, entries); hi; hi = apr_hash_next(hi))
+ {
+ svn_fs_dirent_t *dirent;
+ const char *sub_path;
+ svn_pool_clear(iterpool);
+
+ dirent = apr_hash_this_val(hi);
+
+ /* Skip directories if we want to report files only. */
+ if (dirent->kind == svn_node_dir && depth == svn_depth_files)
+ continue;
+
+ /* Skip paths that we don't have access to? */
+ sub_path = svn_dirent_join(path, dirent->name, iterpool);
+ if (authz_read_func)
+ {
+ svn_boolean_t has_access;
+ SVN_ERR(authz_read_func(&has_access, root, path, authz_read_baton,
+ iterpool));
+ if (!has_access)
+ continue;
+ }
+
+ /* Report entry, if it passes the filter. */
+ SVN_ERR(report_dirent(root, sub_path, dirent->kind, dirent->name,
+ pattern, path_info_only,
+ receiver, receiver_baton, iterpool));
+
+ /* Check for cancellation before recursing down. This should be
+ * slightly more responsive for deep trees. */
+ if (cancel_func)
+ SVN_ERR(cancel_func(cancel_baton));
+
+ /* Recurse on directories. */
+ if (depth == svn_depth_infinity && dirent->kind == svn_node_dir)
+ SVN_ERR(do_list(root, sub_path, pattern, svn_depth_infinity,
+ path_info_only, authz_read_func, authz_read_baton,
+ receiver, receiver_baton, cancel_func,
+ cancel_baton, iterpool));
+ }
+
+ svn_pool_destroy(iterpool);
+
+ return SVN_NO_ERROR;
+}
+
+svn_error_t *
+svn_repos_list(svn_fs_root_t *root,
+ const char *path,
+ const char *pattern,
+ svn_depth_t depth,
+ svn_boolean_t path_info_only,
+ svn_repos_authz_func_t authz_read_func,
+ void *authz_read_baton,
+ svn_repos_dirent_receiver_t receiver,
+ void *receiver_baton,
+ svn_cancel_func_t cancel_func,
+ void *cancel_baton,
+ apr_pool_t *pool)
+{
+ /* Parameter check. */
+ svn_node_kind_t kind;
+ if (depth < svn_depth_empty)
+ return svn_error_createf(SVN_ERR_REPOS_BAD_ARGS, NULL,
+ "Invalid depth '%d' in svn_repos_list", depth);
+
+ /* Do we have access this sub-tree? */
+ if (authz_read_func)
+ {
+ svn_boolean_t has_access;
+ SVN_ERR(authz_read_func(&has_access, root, path, authz_read_baton,
+ pool));
+ if (!has_access)
+ return SVN_NO_ERROR;
+ }
+
+ /* Does the sub-tree even exist?
+ *
+ * Note that we must do this after the authz check to not indirectly
+ * confirm the existence of PATH. */
+ SVN_ERR(svn_fs_check_path(&kind, root, path, pool));
+ if (kind != svn_node_dir)
+ return svn_error_createf(SVN_ERR_FS_NOT_DIRECTORY, NULL,
+ "There is no directory '%s'", path);
+
+ /* Actually report PATH, if it passes the filters. */
+ SVN_ERR(report_dirent(root, path, kind, svn_dirent_dirname(path, pool),
+ pattern, path_info_only, receiver, receiver_baton,
+ pool));
+
+ /* Report directory contents if requested. */
+ if (depth > svn_depth_empty)
+ SVN_ERR(do_list(root, path, pattern, svn_depth_infinity,
+ path_info_only, authz_read_func, authz_read_baton,
+ receiver, receiver_baton, cancel_func, cancel_baton,
+ pool));
+
+ return SVN_NO_ERROR;
+}
Propchange: subversion/trunk/subversion/libsvn_repos/list.c
------------------------------------------------------------------------------
svn:eol-style = native
Modified: subversion/trunk/subversion/libsvn_repos/repos.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_repos/repos.c?rev=1765088&r1=1765087&r2=1765088&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_repos/repos.c (original)
+++ subversion/trunk/subversion/libsvn_repos/repos.c Sat Oct 15 15:58:22 2016
@@ -2061,45 +2061,6 @@ svn_repos_version(void)
SVN_VERSION_BODY;
}
-
-
-svn_error_t *
-svn_repos_stat(svn_dirent_t **dirent,
- svn_fs_root_t *root,
- const char *path,
- apr_pool_t *pool)
-{
- svn_node_kind_t kind;
- svn_dirent_t *ent;
- const char *datestring;
-
- SVN_ERR(svn_fs_check_path(&kind, root, path, pool));
-
- if (kind == svn_node_none)
- {
- *dirent = NULL;
- return SVN_NO_ERROR;
- }
-
- ent = svn_dirent_create(pool);
- ent->kind = kind;
-
- if (kind == svn_node_file)
- SVN_ERR(svn_fs_file_length(&(ent->size), root, path, pool));
-
- SVN_ERR(svn_fs_node_has_props(&ent->has_props, root, path, pool));
-
- SVN_ERR(svn_repos_get_committed_info(&(ent->created_rev),
- &datestring,
- &(ent->last_author),
- root, path, pool));
- if (datestring)
- SVN_ERR(svn_time_from_cstring(&(ent->time), datestring, pool));
-
- *dirent = ent;
- return SVN_NO_ERROR;
-}
-
svn_error_t *
svn_repos_remember_client_capabilities(svn_repos_t *repos,
const apr_array_header_t *capabilities)
Modified: subversion/trunk/subversion/tests/libsvn_repos/repos-test.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/libsvn_repos/repos-test.c?rev=1765088&r1=1765087&r2=1765088&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/libsvn_repos/repos-test.c (original)
+++ subversion/trunk/subversion/tests/libsvn_repos/repos-test.c Sat Oct 15 15:58:22 2016
@@ -3882,6 +3882,51 @@ commit_aborted_txn(const svn_test_opts_t
return SVN_NO_ERROR;
}
+static svn_error_t *
+list_callback(const char *path,
+ svn_dirent_t *dirent,
+ void *baton,
+ apr_pool_t *pool)
+{
+ *(int *)baton += 1;
+
+ return SVN_NO_ERROR;
+}
+
+
+static svn_error_t *
+test_list(const svn_test_opts_t *opts,
+ apr_pool_t *pool)
+{
+ svn_repos_t *repos;
+ svn_fs_t *fs;
+ svn_fs_txn_t *txn;
+ svn_fs_root_t *txn_root, *rev_root;
+ svn_revnum_t youngest_rev;
+ int counter = 0;
+
+ /* Create yet another greek tree repository. */
+ SVN_ERR(svn_test__create_repos(&repos, "test-repo-list", opts, pool));
+ fs = svn_repos_fs(repos);
+
+ /* Prepare a txn to receive the greek tree. */
+ SVN_ERR(svn_fs_begin_txn(&txn, fs, 0, pool));
+ SVN_ERR(svn_fs_txn_root(&txn_root, txn, pool));
+ SVN_ERR(svn_test__create_greek_tree(txn_root, pool));
+ SVN_ERR(svn_repos_fs_commit_txn(NULL, repos, &youngest_rev, txn, pool));
+ SVN_TEST_ASSERT(SVN_IS_VALID_REVNUM(youngest_rev));
+
+ /* List all nodes under /A that contain an 'a'. */
+
+ SVN_ERR(svn_fs_revision_root(&rev_root, fs, youngest_rev, pool));
+ SVN_ERR(svn_repos_list(rev_root, "/A", "*a*", svn_depth_infinity, FALSE,
+ NULL, NULL, list_callback, &counter, NULL, NULL,
+ pool));
+ SVN_TEST_ASSERT(counter == 6);
+
+ return SVN_NO_ERROR;
+}
+
/* The test table. */
static int max_threads = 4;
@@ -3941,6 +3986,8 @@ static struct svn_test_descriptor_t test
"authz for svn_repos_trace_node_locations"),
SVN_TEST_OPTS_PASS(commit_aborted_txn,
"test committing a previously aborted txn"),
+ SVN_TEST_OPTS_PASS(test_list,
+ "test svn_repos_list"),
SVN_TEST_NULL
};
Re: svn commit: r1765088 - in /subversion/trunk/subversion:
include/svn_repos.h libsvn_repos/list.c libsvn_repos/repos.c
tests/libsvn_repos/repos-test.c
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
stefan2@apache.org wrote on Sat, Oct 15, 2016 at 15:58:22 -0000:
> +/**
> + * Efficiently list everything within a sub-tree. Specify a glob pattern
> + * to search for specific files and folders.
> + *
> + * Walk the sub-tree starting at @a path under @a root up to the given
> + * @a depth. For each directory entry found, @a receiver will be called
> + * with @a receiver_baton.
> + *
> + * @a path must point to a directory and @a depth must be at least
> + * @c svn_depth_empty.
> + *
> + * Use @a pool for temporary memory allocation.
> + *
> + * @since New in 1.10.
> + */
> +++ subversion/trunk/subversion/libsvn_repos/list.c Sat Oct 15 15:58:22 2016
> @@ -0,0 +1,248 @@
> +svn_error_t *
> +svn_repos_list(svn_fs_root_t *root,
> + const char *path,
> + const char *pattern,
> + svn_depth_t depth,
> + svn_boolean_t path_info_only,
> + svn_repos_authz_func_t authz_read_func,
> + void *authz_read_baton,
> + svn_repos_dirent_receiver_t receiver,
> + void *receiver_baton,
> + svn_cancel_func_t cancel_func,
> + void *cancel_baton,
> + apr_pool_t *pool)
> +{
> + /* Report directory contents if requested. */
> + if (depth > svn_depth_empty)
> + SVN_ERR(do_list(root, path, pattern, svn_depth_infinity,
If depth == svn_depth_files or depth == svn_depth_immediates, the
recursive call is done with depth infinity. That doesn't look right...
> + path_info_only, authz_read_func, authz_read_baton,
> + receiver, receiver_baton, cancel_func, cancel_baton,
> + pool));
> +
> + return SVN_NO_ERROR;
> +}
>
> Propchange: subversion/trunk/subversion/libsvn_repos/list.c
> ------------------------------------------------------------------------------
> svn:eol-style = native
>
Re: svn commit: r1765088 - in /subversion/trunk/subversion:
include/svn_repos.h libsvn_repos/list.c libsvn_repos/repos.c tests/libsvn_repos/repos-test.c
Posted by Johan Corveleyn <jc...@gmail.com>.
On Sat, Oct 15, 2016 at 5:58 PM, <st...@apache.org> wrote:
> Author: stefan2
> Date: Sat Oct 15 15:58:22 2016
> New Revision: 1765088
>
> URL: http://svn.apache.org/viewvc?rev=1765088&view=rev
> Log:
> Introduce a "directory list" operation at the repos layer. This is the
> first step towards an efficient implementation of 'svn list -r'.
>
> The function is straight-forward with support for operation depth and
> authz. One important use-case is "find files". To support this with
> minimal system load, the function allows to specify a glob pattern for
> the file name and has an option to only report path and node type.
w00t!
/me likes :-)
Thanks for working on this, Stefan.
--
Johan
Re: svn commit: r1765088 - in /subversion/trunk/subversion:
include/svn_repos.h libsvn_repos/list.c libsvn_repos/repos.c
tests/libsvn_repos/repos-test.c
Posted by Daniel Shahaf <d....@daniel.shahaf.name>.
stefan2@apache.org wrote on Sat, Oct 15, 2016 at 15:58:22 -0000:
> +/**
> + * Efficiently list everything within a sub-tree. Specify a glob pattern
> + * to search for specific files and folders.
> + *
> + * Walk the sub-tree starting at @a path under @a root up to the given
> + * @a depth. For each directory entry found, @a receiver will be called
> + * with @a receiver_baton.
> + *
> + * @a path must point to a directory and @a depth must be at least
> + * @c svn_depth_empty.
> + *
> + * Use @a pool for temporary memory allocation.
> + *
> + * @since New in 1.10.
> + */
> +++ subversion/trunk/subversion/libsvn_repos/list.c Sat Oct 15 15:58:22 2016
> @@ -0,0 +1,248 @@
> +svn_error_t *
> +svn_repos_list(svn_fs_root_t *root,
> + const char *path,
> + const char *pattern,
> + svn_depth_t depth,
> + svn_boolean_t path_info_only,
> + svn_repos_authz_func_t authz_read_func,
> + void *authz_read_baton,
> + svn_repos_dirent_receiver_t receiver,
> + void *receiver_baton,
> + svn_cancel_func_t cancel_func,
> + void *cancel_baton,
> + apr_pool_t *pool)
> +{
> + /* Report directory contents if requested. */
> + if (depth > svn_depth_empty)
> + SVN_ERR(do_list(root, path, pattern, svn_depth_infinity,
If depth == svn_depth_files or depth == svn_depth_immediates, the
recursive call is done with depth infinity. That doesn't look right...
> + path_info_only, authz_read_func, authz_read_baton,
> + receiver, receiver_baton, cancel_func, cancel_baton,
> + pool));
> +
> + return SVN_NO_ERROR;
> +}
>
> Propchange: subversion/trunk/subversion/libsvn_repos/list.c
> ------------------------------------------------------------------------------
> svn:eol-style = native
>