You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by st...@apache.org on 2011/02/19 11:47:17 UTC
svn commit: r1072302 - in /subversion/trunk: ./ subversion/include/private/
subversion/libsvn_fs_fs/ subversion/libsvn_subr/
subversion/tests/libsvn_subr/
Author: stefan2
Date: Sat Feb 19 10:47:16 2011
New Revision: 1072302
URL: http://svn.apache.org/viewvc?rev=1072302&view=rev
Log:
Merge all changes (r1068724, r1068739) from the
integrate-cache-item-serialization branch.
These patches introduce a very simple, limited purpose
serialization framework and use that to switch the
caching API from data copying to the more generally
applicable (de-)serialization mechanism.
This makes all types of caches usable with all types
of cacheable items.
Added:
subversion/trunk/subversion/include/private/svn_temp_serializer.h
- copied unchanged from r1068739, subversion/branches/integrate-cache-item-serialization/subversion/include/private/svn_temp_serializer.h
subversion/trunk/subversion/libsvn_fs_fs/temp_serializer.c
- copied unchanged from r1068739, subversion/branches/integrate-cache-item-serialization/subversion/libsvn_fs_fs/temp_serializer.c
subversion/trunk/subversion/libsvn_fs_fs/temp_serializer.h
- copied unchanged from r1068739, subversion/branches/integrate-cache-item-serialization/subversion/libsvn_fs_fs/temp_serializer.h
subversion/trunk/subversion/libsvn_subr/svn_temp_serializer.c
- copied unchanged from r1068739, subversion/branches/integrate-cache-item-serialization/subversion/libsvn_subr/svn_temp_serializer.c
Modified:
subversion/trunk/ (props changed)
subversion/trunk/subversion/include/private/svn_cache.h
subversion/trunk/subversion/libsvn_fs_fs/caching.c
subversion/trunk/subversion/libsvn_fs_fs/dag.c
subversion/trunk/subversion/libsvn_fs_fs/dag.h
subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c
subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h
subversion/trunk/subversion/libsvn_fs_fs/id.c
subversion/trunk/subversion/libsvn_fs_fs/id.h
subversion/trunk/subversion/libsvn_fs_fs/tree.c
subversion/trunk/subversion/libsvn_subr/cache-inprocess.c
subversion/trunk/subversion/libsvn_subr/cache-memcache.c
subversion/trunk/subversion/tests/libsvn_subr/cache-test.c
Propchange: subversion/trunk/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Sat Feb 19 10:47:16 2011
@@ -13,6 +13,7 @@
/subversion/branches/gnome-keyring:870558-871410
/subversion/branches/http-protocol-v2:874395-876041
/subversion/branches/in-memory-cache:869829-871452
+/subversion/branches/integrate-cache-item-serialization:1068724-1068739
/subversion/branches/integrate-cache-membuffer:998649-998852
/subversion/branches/integrate-compression-level:1068651-1072287
/subversion/branches/integrate-io-improvements:1068684-1072297
@@ -29,7 +30,7 @@
/subversion/branches/log-g-performance:870941-871032
/subversion/branches/merge-skips-obstructions:874525-874615
/subversion/branches/nfc-nfd-aware-client:870276,870376
-/subversion/branches/performance:979193,980118,981087,981287,981684,981827,982043,982355,983764,983766,984927,984973,984984,985014,985037,985046,985472,985477,985482,985487-985488,985493,985497,985500,985606,985669,986453,986465,986491,986517,987888,987893,988319,990541,990568,990572,990600,990759,992899,992911,993127,993141,994956,995478,995507,995603,998012,998858,999098,1001413,1001417,1004291,1022668,1022670,1022676,1022715,1022719,1025660,1025672,1027193,1027203,1027206,1027214,1027227,1028077,1028092,1028094,1028104,1028107,1028111,1028354,1029038,1029042-1029043,1029062,1029078,1029080,1029090,1029092-1029093,1029111,1029151,1029158,1029232,1029335,1029339-1029340,1029342,1029344,1030763,1030827,1031203,1031235,1032285,1032333,1033040,1033057,1033294,1035869,1039511,1043705,1053735,1056015,1066452,1067683
+/subversion/branches/performance:979193,980118,981087,981090,981189,981194,981287,981684,981827,982043,982355,983398,983406,983474,983488,983490,983764,983766,984927,984973,984984,985014,985037,985046,985472,985477,985482,985487-985488,985493,985497,985500,985603,985606,985669,986453,986465,986491,986517,986521,986608,986832,987865,987868-987869,987872,987886,987888,987893,988319,988898,990533,990535-990536,990541,990568,990572,990600,990759,992899,992911,993127,993141,994956,995478,995507,995603,998012,998858,999098,1001413,1001417,1004291,1022668,1022670,1022676,1022715,1022719,1025660,1025672,1027193,1027203,1027206,1027214,1027227,1028077,1028092,1028094,1028104,1028107,1028111,1028354,1029038,1029042-1029043,1029062,1029078,1029080,1029090,1029092-1029093,1029111,1029151,1029158,1029229,1029232,1029335,1029339-1029340,1029342,1029344,1030763,1030827,1031203,1031235,1032285,1032333,1033040,1033057,1033294,1035869,1039511,1043705,1053735,1056015,1066452,1067683
/subversion/branches/py-tests-as-modules:956579-1033052
/subversion/branches/ra_serf-digest-authn:875693-876404
/subversion/branches/reintegrate-improvements:873853-874164
Modified: subversion/trunk/subversion/include/private/svn_cache.h
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/include/private/svn_cache.h?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/include/private/svn_cache.h (original)
+++ subversion/trunk/subversion/include/private/svn_cache.h Sat Feb 19 10:47:16 2011
@@ -120,7 +120,7 @@ typedef struct svn_cache__t svn_cache__t
* for all of its storage needs. The elements in the cache will be
* indexed by keys of length @a klen, which may be APR_HASH_KEY_STRING
* if they are strings. Cached values will be copied in and out of
- * the cache using @a dup_func.
+ * the cache using @a serialize_func and @a deserialize_func, respectively.
*
* The cache stores up to @a pages * @a items_per_page items at a
* time. The exact cache invalidation strategy is not defined here,
@@ -140,7 +140,8 @@ typedef struct svn_cache__t svn_cache__t
*/
svn_error_t *
svn_cache__create_inprocess(svn_cache__t **cache_p,
- svn_cache__dup_func_t dup_func,
+ svn_cache__serialize_func_t serialize_func,
+ svn_cache__deserialize_func_t deserialize_func,
apr_ssize_t klen,
apr_int64_t pages,
apr_int64_t items_per_page,
@@ -207,7 +208,7 @@ svn_cache__make_memcache_from_config(svn
*
* Allocations will be made in @a pool, in particular the data buffers.
*/
-svn_error_t*
+svn_error_t *
svn_cache__membuffer_cache_create(svn_membuffer_t **cache,
apr_size_t total_size,
apr_size_t directory_size,
Modified: subversion/trunk/subversion/libsvn_fs_fs/caching.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/caching.c?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/caching.c (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/caching.c Sat Feb 19 10:47:16 2011
@@ -24,138 +24,13 @@
#include "fs_fs.h"
#include "id.h"
#include "dag.h"
+#include "temp_serializer.h"
#include "../libsvn_fs/fs-loader.h"
#include "svn_config.h"
#include "svn_private_config.h"
-/*** Dup/serialize/deserialize functions. ***/
-
-
-/** Caching SVN_FS_ID_T values. **/
-/* Implements svn_cache__dup_func_t */
-static svn_error_t *
-dup_id(void **out,
- const void *in,
- apr_pool_t *pool)
-{
- const svn_fs_id_t *id = in;
- *out = svn_fs_fs__id_copy(id, pool);
- return SVN_NO_ERROR;
-}
-
-/* Implements svn_cache__serialize_func_t */
-static svn_error_t *
-serialize_id(char **data,
- apr_size_t *data_len,
- void *in,
- apr_pool_t *pool)
-{
- svn_fs_id_t *id = in;
- svn_string_t *id_str = svn_fs_fs__id_unparse(id, pool);
- *data = (char *) id_str->data;
- *data_len = id_str->len;
-
- return SVN_NO_ERROR;
-}
-
-
-/* Implements svn_cache__deserialize_func_t */
-static svn_error_t *
-deserialize_id(void **out,
- const char *data,
- apr_size_t data_len,
- apr_pool_t *pool)
-{
- svn_fs_id_t *id = svn_fs_fs__id_parse(data, data_len, pool);
- if (id == NULL)
- {
- return svn_error_create(SVN_ERR_FS_NOT_ID, NULL,
- _("Bad ID in cache"));
- }
-
- *out = id;
- return SVN_NO_ERROR;
-}
-
-
-/** Caching directory listings. **/
-/* Implements svn_cache__dup_func_t */
-static svn_error_t *
-dup_dir_listing(void **out,
- const void *in,
- apr_pool_t *pool)
-{
- apr_hash_t *new_entries = apr_hash_make(pool);
- apr_hash_t *entries = (void*)in; /* Cast away const only */
- apr_hash_index_t *hi;
-
- for (hi = apr_hash_first(pool, entries); hi; hi = apr_hash_next(hi))
- {
- svn_fs_dirent_t *dirent = svn__apr_hash_index_val(hi);
- svn_fs_dirent_t *new_dirent;
-
- new_dirent = apr_palloc(pool, sizeof(*new_dirent));
- new_dirent->name = apr_pstrdup(pool, dirent->name);
- new_dirent->kind = dirent->kind;
- new_dirent->id = svn_fs_fs__id_copy(dirent->id, pool);
- apr_hash_set(new_entries, new_dirent->name, APR_HASH_KEY_STRING,
- new_dirent);
- }
-
- *out = new_entries;
- return SVN_NO_ERROR;
-}
-
-
-/** Caching packed rev offsets. **/
-/* Implements svn_cache__serialize_func_t */
-static svn_error_t *
-manifest_serialize(char **data,
- apr_size_t *data_len,
- void *in,
- apr_pool_t *pool)
-{
- apr_array_header_t *manifest = in;
-
- *data_len = sizeof(apr_off_t) * manifest->nelts;
- *data = apr_palloc(pool, *data_len);
- memcpy(*data, manifest->elts, *data_len);
-
- return SVN_NO_ERROR;
-}
-
-/* Implements svn_cache__deserialize_func_t */
-static svn_error_t *
-manifest_deserialize(void **out,
- const char *data,
- apr_size_t data_len,
- apr_pool_t *pool)
-{
- apr_array_header_t *manifest = apr_array_make(pool,
- (int) (data_len / sizeof(apr_off_t)),
- sizeof(apr_off_t));
- memcpy(manifest->elts, data, data_len);
- manifest->nelts = (int) (data_len / sizeof(apr_off_t));
- *out = manifest;
-
- return SVN_NO_ERROR;
-}
-
-/* Implements svn_cache__dup_func_t */
-static svn_error_t *
-dup_pack_manifest(void **out,
- const void *in,
- apr_pool_t *pool)
-{
- const apr_array_header_t *manifest = in;
-
- *out = apr_array_copy(pool, manifest);
- return SVN_NO_ERROR;
-}
-
-
/* Return a memcache in *MEMCACHE_P for FS if it's configured to use
memcached, or NULL otherwise. Also, sets *FAIL_STOP to a boolean
indicating whether cache errors should be returned to the caller or
@@ -212,38 +87,41 @@ svn_fs_fs__initialize_caches(svn_fs_t *f
* id_private_t + 3 strings for value, and the cache_entry); the
* default pool size is 8192, so about a hundred should fit
* comfortably. */
- if (memcache)
- SVN_ERR(svn_cache__create_memcache(&(ffd->rev_root_id_cache),
- memcache,
- serialize_id,
- deserialize_id,
- sizeof(svn_revnum_t),
- apr_pstrcat(pool, prefix, "RRI",
- (char *)NULL),
- fs->pool));
+ if (svn_fs__get_global_membuffer_cache())
+ SVN_ERR(svn_cache__create_membuffer_cache(&(ffd->rev_root_id_cache),
+ svn_fs__get_global_membuffer_cache(),
+ svn_fs_fs__serialize_id,
+ svn_fs_fs__deserialize_id,
+ sizeof(svn_revnum_t),
+ apr_pstrcat(pool, prefix, "RRI",
+ (char *)NULL),
+ fs->pool));
else
- SVN_ERR(svn_cache__create_inprocess(&(ffd->rev_root_id_cache),
- dup_id, sizeof(svn_revnum_t),
- 1, 100, FALSE, fs->pool));
+ SVN_ERR(svn_cache__create_inprocess(&(ffd->rev_root_id_cache),
+ svn_fs_fs__serialize_id,
+ svn_fs_fs__deserialize_id,
+ sizeof(svn_revnum_t),
+ 1, 100, FALSE, fs->pool));
if (! no_handler)
- SVN_ERR(svn_cache__set_error_handler(ffd->rev_root_id_cache,
- warn_on_cache_errors, fs, pool));
+ SVN_ERR(svn_cache__set_error_handler(ffd->rev_root_id_cache,
+ warn_on_cache_errors, fs, pool));
/* Rough estimate: revision DAG nodes have size around 320 bytes, so
* let's put 16 on a page. */
- if (memcache)
- SVN_ERR(svn_cache__create_memcache(&(ffd->rev_node_cache),
- memcache,
- svn_fs_fs__dag_serialize,
- svn_fs_fs__dag_deserialize,
- APR_HASH_KEY_STRING,
- apr_pstrcat(pool, prefix, "DAG",
- (char *)NULL),
- fs->pool));
+ if (svn_fs__get_global_membuffer_cache())
+ SVN_ERR(svn_cache__create_membuffer_cache(&(ffd->rev_node_cache),
+ svn_fs__get_global_membuffer_cache(),
+ svn_fs_fs__dag_serialize,
+ svn_fs_fs__dag_deserialize,
+ APR_HASH_KEY_STRING,
+ apr_pstrcat(pool, prefix, "DAG",
+ (char *)NULL),
+ fs->pool));
else
SVN_ERR(svn_cache__create_inprocess(&(ffd->rev_node_cache),
- svn_fs_fs__dag_dup_for_cache,
+ svn_fs_fs__dag_serialize,
+ svn_fs_fs__dag_deserialize,
APR_HASH_KEY_STRING,
1024, 16, FALSE, fs->pool));
if (! no_handler)
@@ -252,18 +130,20 @@ svn_fs_fs__initialize_caches(svn_fs_t *f
/* Very rough estimate: 1K per directory. */
- if (memcache)
- SVN_ERR(svn_cache__create_memcache(&(ffd->dir_cache),
- memcache,
- svn_fs_fs__dir_entries_serialize,
- svn_fs_fs__dir_entries_deserialize,
- APR_HASH_KEY_STRING,
- apr_pstrcat(pool, prefix, "DIR",
- (char *)NULL),
- fs->pool));
+ if (svn_fs__get_global_membuffer_cache())
+ SVN_ERR(svn_cache__create_membuffer_cache(&(ffd->dir_cache),
+ svn_fs__get_global_membuffer_cache(),
+ svn_fs_fs__serialize_dir_entries,
+ svn_fs_fs__deserialize_dir_entries,
+ APR_HASH_KEY_STRING,
+ apr_pstrcat(pool, prefix, "DIR",
+ (char *)NULL),
+ fs->pool));
else
SVN_ERR(svn_cache__create_inprocess(&(ffd->dir_cache),
- dup_dir_listing, APR_HASH_KEY_STRING,
+ svn_fs_fs__serialize_dir_entries,
+ svn_fs_fs__deserialize_dir_entries,
+ APR_HASH_KEY_STRING,
1024, 8, FALSE, fs->pool));
if (! no_handler)
@@ -272,24 +152,27 @@ svn_fs_fs__initialize_caches(svn_fs_t *f
/* Only 16 bytes per entry (a revision number + the corresponding offset).
Since we want ~8k pages, that means 512 entries per page. */
- if (memcache)
- SVN_ERR(svn_cache__create_memcache(&(ffd->packed_offset_cache),
- memcache,
- manifest_serialize,
- manifest_deserialize,
- sizeof(svn_revnum_t),
- apr_pstrcat(pool, prefix, "PACK-MANIFEST",
- (char *)NULL),
- fs->pool));
+ if (svn_fs__get_global_membuffer_cache())
+ SVN_ERR(svn_cache__create_membuffer_cache(&(ffd->packed_offset_cache),
+ svn_fs__get_global_membuffer_cache(),
+ svn_fs_fs__serialize_manifest,
+ svn_fs_fs__deserialize_manifest,
+ sizeof(svn_revnum_t),
+ apr_pstrcat(pool, prefix, "PACK-MANIFEST",
+ (char *)NULL),
+ fs->pool));
else
SVN_ERR(svn_cache__create_inprocess(&(ffd->packed_offset_cache),
- dup_pack_manifest, sizeof(svn_revnum_t),
+ svn_fs_fs__serialize_manifest,
+ svn_fs_fs__deserialize_manifest,
+ sizeof(svn_revnum_t),
32, 1, FALSE, fs->pool));
if (! no_handler)
SVN_ERR(svn_cache__set_error_handler(ffd->packed_offset_cache,
warn_on_cache_errors, fs, pool));
+ /* initialize fulltext cache as configured */
if (memcache)
{
SVN_ERR(svn_cache__create_memcache(&(ffd->fulltext_cache),
Modified: subversion/trunk/subversion/libsvn_fs_fs/dag.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/dag.c?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/dag.c (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/dag.c Sat Feb 19 10:47:16 2011
@@ -38,6 +38,8 @@
#include "private/svn_fspath.h"
#include "svn_private_config.h"
+#include "private/svn_temp_serializer.h"
+#include "temp_serializer.h"
/* Initializing a filesystem. */
@@ -1056,59 +1058,38 @@ svn_fs_fs__dag_dup(const dag_node_t *nod
}
svn_error_t *
-svn_fs_fs__dag_dup_for_cache(void **out,
- const void *in,
- apr_pool_t *pool)
-{
- const dag_node_t *in_node = in;
- dag_node_t *out_node;
-
- out_node = svn_fs_fs__dag_dup(in_node, pool);
- out_node->fs = NULL;
- *out = out_node;
- return SVN_NO_ERROR;
-}
-
-/* The cache serialization format is:
- *
- * - For mutable nodes: the character 'M', then 'F' for files or 'D'
- * for directories, then the ID, then '\n', then the created path.
- *
- * - For immutable nodes: the character 'I' followed by the noderev
- * hash dump (the other fields can be reconstructed from this). (We
- * assume that, once constructed, immutable nodes always contain
- * their noderev.)
- */
-
-svn_error_t *
svn_fs_fs__dag_serialize(char **data,
apr_size_t *data_len,
void *in,
apr_pool_t *pool)
{
dag_node_t *node = in;
- svn_stringbuf_t *buf = svn_stringbuf_create("", pool);
+ svn_stringbuf_t *serialized;
- if (svn_fs_fs__dag_check_mutable(node))
- {
- svn_stringbuf_appendcstr(buf, "M");
- svn_stringbuf_appendcstr(buf, (node->kind == svn_node_file ? "F" : "D"));
- svn_stringbuf_appendcstr(buf, svn_fs_fs__id_unparse(node->id,
- pool)->data);
- svn_stringbuf_appendcstr(buf, "\n");
- svn_stringbuf_appendcstr(buf, node->created_path);
- }
+ /* create an serialization context and serialize the dag node as root */
+ svn_temp_serializer__context_t *context =
+ svn_temp_serializer__init(node,
+ sizeof(*node),
+ 503,
+ pool);
+
+ /* for mutable nodes, we will _never_ cache the noderev */
+ if (node->node_revision && !svn_fs_fs__dag_check_mutable(node))
+ svn_fs_fs__noderev_serialize(context, &node->node_revision);
else
- {
- fs_fs_data_t *ffd = node->fs->fsap_data;
- svn_stringbuf_appendcstr(buf, "I");
- SVN_ERR(svn_fs_fs__write_noderev(svn_stream_from_stringbuf(buf, pool),
- node->node_revision, ffd->format,
- TRUE, pool));
- }
+ svn_temp_serializer__set_null(context,
+ (const void * const *)&node->node_revision);
+
+ /* serialize other sub-structures */
+ svn_fs_fs__id_serialize(context, (const svn_fs_id_t **)&node->id);
+ svn_fs_fs__id_serialize(context, &node->fresh_root_predecessor_id);
+ svn_temp_serializer__add_string(context, &node->created_path);
+
+ /* return serialized data */
+ serialized = svn_temp_serializer__get(context);
+ *data = serialized->data;
+ *data_len = serialized->len;
- *data = buf->data;
- *data_len = buf->len;
return SVN_NO_ERROR;
}
@@ -1118,74 +1099,23 @@ svn_fs_fs__dag_deserialize(void **out,
apr_size_t data_len,
apr_pool_t *pool)
{
- dag_node_t *node = apr_pcalloc(pool, sizeof(*node));
-
+ dag_node_t *node = (dag_node_t *)data;
if (data_len == 0)
return svn_error_create(SVN_ERR_FS_CORRUPT, NULL,
_("Empty noderev in cache"));
- if (*data == 'M')
- {
- const char *newline;
- size_t id_len;
-
- data++; data_len--;
- if (data_len == 0)
- return svn_error_create(SVN_ERR_FS_CORRUPT, NULL,
- _("Kindless noderev in cache"));
- if (*data == 'F')
- node->kind = svn_node_file;
- else if (*data == 'D')
- node->kind = svn_node_dir;
- else
- return svn_error_createf(SVN_ERR_FS_CORRUPT, NULL,
- _("Unknown kind for noderev in cache: '%c'"),
- *data);
-
- data++; data_len--;
- newline = memchr(data, '\n', data_len);
- if (!newline)
- return svn_error_create(SVN_ERR_FS_CORRUPT, NULL,
- _("Unterminated ID in cache"));
- id_len = newline - 1 - data;
- node->id = svn_fs_fs__id_parse(data, id_len, pool);
- if (! node->id)
- return svn_error_createf(SVN_ERR_FS_CORRUPT, NULL,
- _("Bogus ID '%s' in cache"),
- apr_pstrndup(pool, data, id_len));
-
- data += id_len; data_len -= id_len;
- data++; data_len--;
- if (data_len == 0)
- return svn_error_create(SVN_ERR_FS_CORRUPT, NULL,
- _("No created path"));
- node->created_path = apr_pstrndup(pool, data, data_len);
- }
- else if (*data == 'I')
- {
- node_revision_t *noderev;
- apr_pool_t *subpool = svn_pool_create(pool);
- svn_stream_t *stream =
- svn_stream_from_stringbuf(svn_stringbuf_ncreate(data + 1,
- data_len - 1,
- subpool),
- subpool);
- SVN_ERR(svn_fs_fs__read_noderev(&noderev, stream, pool));
- node->kind = noderev->kind;
- node->id = svn_fs_fs__id_copy(noderev->id, pool);
- node->created_path = apr_pstrdup(pool, noderev->created_path);
-
- if (noderev->is_fresh_txn_root)
- node->fresh_root_predecessor_id = noderev->predecessor_id;
+ /* Copy the _full_ buffer as it also contains the sub-structures. */
+ node->fs = NULL;
- node->node_revision = noderev;
+ /* fixup all references to sub-structures */
+ svn_fs_fs__id_deserialize(node, &node->id);
+ svn_fs_fs__id_deserialize(node,
+ (svn_fs_id_t **)&node->fresh_root_predecessor_id);
+ svn_fs_fs__noderev_deserialize(node, &node->node_revision);
- svn_pool_destroy(subpool);
- }
- else
- return svn_error_createf(SVN_ERR_FS_CORRUPT, NULL,
- _("Unknown node type in cache: '%c'"), *data);
+ svn_temp_deserializer__resolve(node, (void**)&node->created_path);
+ /* return result */
*out = node;
return SVN_NO_ERROR;
Modified: subversion/trunk/subversion/libsvn_fs_fs/dag.h
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/dag.h?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/dag.h (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/dag.h Sat Feb 19 10:47:16 2011
@@ -78,13 +78,6 @@ dag_node_t *
svn_fs_fs__dag_dup(const dag_node_t *node,
apr_pool_t *pool);
-/* Like svn_fs_fs__dag_dup, but implementing the svn_cache__dup_func_t
- prototype, and NULLing the FS field. */
-svn_error_t *
-svn_fs_fs__dag_dup_for_cache(void **out,
- const void *in,
- apr_pool_t *pool);
-
/* Serialize a DAG node.
Implements svn_cache__serialize_func_t */
svn_error_t *
Modified: subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/fs_fs.c Sat Feb 19 10:47:16 2011
@@ -3746,26 +3746,6 @@ unparse_dir_entries(apr_hash_t **str_ent
}
-svn_error_t *
-svn_fs_fs__dir_entries_serialize(char **data,
- apr_size_t *data_len,
- void *in,
- apr_pool_t *pool)
-{
- apr_hash_t *entries = in;
- svn_stringbuf_t *buf = svn_stringbuf_create("", pool);
- svn_stream_t *stream = svn_stream_from_stringbuf(buf, pool);
-
- SVN_ERR(unparse_dir_entries(&entries, entries, pool));
- SVN_ERR(svn_hash_write2(entries, stream, SVN_HASH_TERMINATOR, pool));
-
- *data = buf->data;
- *data_len = buf->len;
-
- return SVN_NO_ERROR;
-}
-
-
/* Given a hash STR_ENTRIES with values as svn_string_t as specified
in an FSFS directory contents listing, return a hash of dirents in
*ENTRIES_P. Perform allocations in POOL. */
@@ -3822,24 +3802,6 @@ parse_dir_entries(apr_hash_t **entries_p
}
svn_error_t *
-svn_fs_fs__dir_entries_deserialize(void **out,
- const char *data,
- apr_size_t data_len,
- apr_pool_t *pool)
-{
- apr_hash_t *entries = apr_hash_make(pool);
- svn_stringbuf_t *buf = svn_stringbuf_ncreate(data, data_len, pool);
- svn_stream_t *stream = svn_stream_from_stringbuf(buf, pool);
-
- SVN_ERR(svn_hash_read2(entries, stream, SVN_HASH_TERMINATOR, pool));
- SVN_ERR(parse_dir_entries(&entries, entries, pool));
-
- *out = entries;
- return SVN_NO_ERROR;
-}
-
-
-svn_error_t *
svn_fs_fs__rep_contents_dir(apr_hash_t **entries_p,
svn_fs_t *fs,
node_revision_t *noderev,
Modified: subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/fs_fs.h Sat Feb 19 10:47:16 2011
@@ -99,22 +99,6 @@ svn_error_t *svn_fs_fs__rev_get_root(svn
svn_revnum_t rev,
apr_pool_t *pool);
-/* Serialize a directory contents hash.
- Implements svn_cache__serialize_func_t */
-svn_error_t *
-svn_fs_fs__dir_entries_serialize(char **data,
- apr_size_t *data_len,
- void *in,
- apr_pool_t *pool);
-
-/* Deserialize a directory contents hash.
- Implements svn_cache__deserialize_func_t */
-svn_error_t *
-svn_fs_fs__dir_entries_deserialize(void **out,
- const char *data,
- apr_size_t data_len,
- apr_pool_t *pool);
-
/* Set *ENTRIES to an apr_hash_t of dirent structs that contain the
directory entries of node-revision NODEREV in filesystem FS. The
returned table (and its keys and values) is allocated in POOL,
Modified: subversion/trunk/subversion/libsvn_fs_fs/id.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/id.c?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/id.c (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/id.c Sat Feb 19 10:47:16 2011
@@ -25,7 +25,7 @@
#include "id.h"
#include "../libsvn_fs/fs-loader.h"
-
+#include "private/svn_temp_serializer.h"
typedef struct id_private_t {
@@ -187,6 +187,7 @@ svn_fs_fs__id_txn_create(const char *nod
pvt->txn_id = apr_pstrdup(pool, txn_id);
pvt->rev = SVN_INVALID_REVNUM;
pvt->offset = -1;
+
id->vtable = &id_vtable;
id->fsap_data = pvt;
return id;
@@ -208,6 +209,7 @@ svn_fs_fs__id_rev_create(const char *nod
pvt->txn_id = NULL;
pvt->rev = rev;
pvt->offset = offset;
+
id->vtable = &id_vtable;
id->fsap_data = pvt;
return id;
@@ -226,6 +228,7 @@ svn_fs_fs__id_copy(const svn_fs_id_t *id
new_pvt->txn_id = pvt->txn_id ? apr_pstrdup(pool, pvt->txn_id) : NULL;
new_pvt->rev = pvt->rev;
new_pvt->offset = pvt->offset;
+
new_id->vtable = &id_vtable;
new_id->fsap_data = new_pvt;
return new_id;
@@ -309,3 +312,88 @@ svn_fs_fs__id_parse(const char *data,
return id;
}
+
+/* (de-)serialization support */
+
+/* Serialization of the PVT sub-structure within the CONTEXT.
+ */
+static void
+serialize_id_private(svn_temp_serializer__context_t *context,
+ const id_private_t * const *pvt)
+{
+ const id_private_t *private = *pvt;
+
+ /* serialize the pvt data struct itself */
+ svn_temp_serializer__push(context,
+ (const void * const *)pvt,
+ sizeof(*private));
+
+ /* append the referenced strings */
+ svn_temp_serializer__add_string(context, &private->node_id);
+ svn_temp_serializer__add_string(context, &private->copy_id);
+ svn_temp_serializer__add_string(context, &private->txn_id);
+
+ /* return to caller's nesting level */
+ svn_temp_serializer__pop(context);
+}
+
+/* Serialize an ID within the serialization CONTECT.
+ */
+void
+svn_fs_fs__id_serialize(svn_temp_serializer__context_t *context,
+ const struct svn_fs_id_t * const *id)
+{
+ /* nothing to do for NULL ids */
+ if (*id == NULL)
+ return;
+
+ /* serialize the id data struct itself */
+ svn_temp_serializer__push(context,
+ (const void * const *)id,
+ sizeof(**id));
+
+ /* serialize the id_private_t data sub-struct */
+ serialize_id_private(context,
+ (const id_private_t * const *)&(*id)->fsap_data);
+
+ /* return to caller's nesting level */
+ svn_temp_serializer__pop(context);
+}
+
+/* Deserialization of the PVT sub-structure in BUFFER.
+ */
+static void
+deserialize_id_private(void *buffer, id_private_t **pvt)
+{
+ /* fixup the reference to the only sub-structure */
+ id_private_t *private;
+ svn_temp_deserializer__resolve(buffer, (void**)pvt);
+
+ /* fixup the sub-structure itself */
+ private = *pvt;
+ svn_temp_deserializer__resolve(private, (void**)&private->node_id);
+ svn_temp_deserializer__resolve(private, (void**)&private->copy_id);
+ svn_temp_deserializer__resolve(private, (void**)&private->txn_id);
+}
+
+/* Deserialize an ID inside the BUFFER.
+ */
+void
+svn_fs_fs__id_deserialize(void *buffer, svn_fs_id_t **id)
+{
+ /* The id maybe all what is in the whole buffer.
+ * Don't try to fixup the pointer in that case*/
+ if (*id != buffer)
+ svn_temp_deserializer__resolve(buffer, (void**)id);
+
+ /* no id, no sub-structure fixup necessary */
+ if (*id == NULL)
+ return;
+
+ /* the stored vtable is bogus at best -> set the right one */
+ (*id)->vtable = &id_vtable;
+
+ /* handle sub-structures */
+ deserialize_id_private(*id, (id_private_t **)&(*id)->fsap_data);
+}
+
Modified: subversion/trunk/subversion/libsvn_fs_fs/id.h
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/id.h?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/id.h (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/id.h Sat Feb 19 10:47:16 2011
@@ -90,6 +90,25 @@ svn_fs_id_t *svn_fs_fs__id_parse(const c
apr_size_t len,
apr_pool_t *pool);
+
+/* (de-)serialization support*/
+
+struct svn_temp_serializer__context_t;
+
+/**
+ * Serialize an @a id within the serialization @a context.
+ */
+void
+svn_fs_fs__id_serialize(struct svn_temp_serializer__context_t *context,
+ const svn_fs_id_t * const *id);
+
+/**
+ * Deserialize an @a id within the @a buffer.
+ */
+void
+svn_fs_fs__id_deserialize(void *buffer,
+ svn_fs_id_t **id);
+
#ifdef __cplusplus
}
#endif /* __cplusplus */
Modified: subversion/trunk/subversion/libsvn_fs_fs/tree.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_fs_fs/tree.c?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_fs_fs/tree.c (original)
+++ subversion/trunk/subversion/libsvn_fs_fs/tree.c Sat Feb 19 10:47:16 2011
@@ -3910,9 +3910,10 @@ make_txn_root(svn_fs_root_t **root_p,
Note that since dag_node_cache_invalidate uses svn_cache__iter,
this *cannot* be a memcache-based cache. */
SVN_ERR(svn_cache__create_inprocess(&(frd->txn_node_cache),
- svn_fs_fs__dag_dup_for_cache,
- APR_HASH_KEY_STRING,
- 32, 20, FALSE, root->pool));
+ svn_fs_fs__dag_serialize,
+ svn_fs_fs__dag_deserialize,
+ APR_HASH_KEY_STRING,
+ 32, 20, FALSE, root->pool));
root->fsap_data = frd;
Modified: subversion/trunk/subversion/libsvn_subr/cache-inprocess.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_subr/cache-inprocess.c?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_subr/cache-inprocess.c (original)
+++ subversion/trunk/subversion/libsvn_subr/cache-inprocess.c Sat Feb 19 10:47:16 2011
@@ -37,8 +37,11 @@ typedef struct inprocess_cache_t {
apr_hash_t *hash;
apr_ssize_t klen;
- /* Used to copy values in and out of the cache. */
- svn_cache__dup_func_t dup_func;
+ /* Used to copy values into the cache. */
+ svn_cache__serialize_func_t serialize_func;
+
+ /* Used to copy values out of the cache. */
+ svn_cache__deserialize_func_t deserialize_func;
/* The number of pages we're allowed to allocate before having to
* try to reuse one. */
@@ -94,8 +97,13 @@ struct cache_page {
/* An cache entry. */
struct cache_entry {
const void *key;
+
+ /* serialized value */
void *value;
+ /* length of the serialized value in bytes */
+ apr_size_t size;
+
/* The page it's on (needed so that the LRU list can be
* maintained). */
struct cache_page *page;
@@ -142,21 +150,6 @@ move_page_to_front(inprocess_cache_t *ca
insert_page(cache, page);
}
-/* Uses CACHE->dup_func to copy VALUE into *VALUE_P inside POOL, or
- just sets *VALUE_P to NULL if VALUE is NULL. */
-static svn_error_t *
-duplicate_value(void **value_p,
- inprocess_cache_t *cache,
- void *value,
- apr_pool_t *pool)
-{
- if (value)
- SVN_ERR((cache->dup_func)(value_p, value, pool));
- else
- *value_p = NULL;
- return SVN_NO_ERROR;
-}
-
/* Return a copy of KEY inside POOL, using CACHE->KLEN to figure out
* how. */
static const void *
@@ -214,7 +207,7 @@ inprocess_cache_get(void **value_p,
{
inprocess_cache_t *cache = cache_void;
struct cache_entry *entry;
- svn_error_t *err;
+ char* buffer;
SVN_ERR(lock_cache(cache));
@@ -227,9 +220,23 @@ inprocess_cache_get(void **value_p,
move_page_to_front(cache, entry->page);
+ /* duplicate the buffer entry */
+ buffer = apr_palloc(pool, entry->size);
+ memcpy(buffer, entry->value, entry->size);
+
+ /* the cache is no longer being accessed */
+ SVN_ERR(unlock_cache(cache, SVN_NO_ERROR));
+
+ /* deserialize the buffer content. Usually, this will directly
+ modify the buffer content directly.
+ */
*found = TRUE;
- err = duplicate_value(value_p, cache, entry->value, pool);
- return unlock_cache(cache, err);
+ if (entry->value)
+ return cache->deserialize_func(value_p, buffer, entry->size, pool);
+ else
+ *value_p = NULL;
+
+ return unlock_cache(cache, SVN_NO_ERROR);
}
/* Removes PAGE from the LRU list, removes all of its entries from
@@ -302,8 +309,19 @@ inprocess_cache_set(void *cache_void,
struct cache_page *page = existing_entry->page;
move_page_to_front(cache, page);
- err = duplicate_value(&(existing_entry->value), cache,
- value, page->page_pool);
+ if (value)
+ {
+ err = cache->serialize_func((char **)&existing_entry->value,
+ &existing_entry->size,
+ value,
+ page->page_pool);
+ }
+ else
+ {
+ existing_entry->value = NULL;
+ existing_entry->size = 0;
+ }
+
goto cleanup;
}
@@ -340,8 +358,19 @@ inprocess_cache_set(void *cache_void,
/* Copy the key and value into the page's pool. */
new_entry->key = duplicate_key(cache, key, page->page_pool);
- err = duplicate_value(&(new_entry->value), cache, value,
- page->page_pool);
+ if (value)
+ {
+ err = cache->serialize_func((char **)&new_entry->value,
+ &new_entry->size,
+ value,
+ page->page_pool);
+ }
+ else
+ {
+ new_entry->value = NULL;
+ new_entry->size = 0;
+ }
+
if (err)
goto cleanup;
@@ -417,7 +446,8 @@ static svn_cache__vtable_t inprocess_cac
svn_error_t *
svn_cache__create_inprocess(svn_cache__t **cache_p,
- svn_cache__dup_func_t dup_func,
+ svn_cache__serialize_func_t serialize,
+ svn_cache__deserialize_func_t deserialize,
apr_ssize_t klen,
apr_int64_t pages,
apr_int64_t items_per_page,
@@ -430,7 +460,8 @@ svn_cache__create_inprocess(svn_cache__t
cache->hash = apr_hash_make(pool);
cache->klen = klen;
- cache->dup_func = dup_func;
+ cache->serialize_func = serialize;
+ cache->deserialize_func = deserialize;
SVN_ERR_ASSERT(pages >= 1);
cache->unallocated_pages = pages;
Modified: subversion/trunk/subversion/libsvn_subr/cache-memcache.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_subr/cache-memcache.c?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_subr/cache-memcache.c (original)
+++ subversion/trunk/subversion/libsvn_subr/cache-memcache.c Sat Feb 19 10:47:16 2011
@@ -144,7 +144,7 @@ memcache_get(void **value_p,
SVN_ERR(build_key(&mc_key, cache, key, subpool));
apr_err = apr_memcache_getp(cache->memcache,
- (cache->deserialize_func ? subpool : pool),
+ pool,
mc_key,
&data,
&data_len,
Modified: subversion/trunk/subversion/tests/libsvn_subr/cache-test.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/tests/libsvn_subr/cache-test.c?rev=1072302&r1=1072301&r2=1072302&view=diff
==============================================================================
--- subversion/trunk/subversion/tests/libsvn_subr/cache-test.c (original)
+++ subversion/trunk/subversion/tests/libsvn_subr/cache-test.c Sat Feb 19 10:47:16 2011
@@ -34,22 +34,6 @@
#include "../svn_test.h"
-/* Implements svn_cache__dup_func_t */
-static svn_error_t *
-dup_revnum(void **out,
- const void *in,
- apr_pool_t *pool)
-{
- const svn_revnum_t *in_rn = in;
- svn_revnum_t *duped = apr_palloc(pool, sizeof(*duped));
-
- *duped = *in_rn;
-
- *out = duped;
-
- return SVN_NO_ERROR;
-}
-
/* Implements svn_cache__serialize_func_t */
static svn_error_t *
serialize_revnum(char **data,
@@ -146,12 +130,13 @@ test_inprocess_cache_basic(apr_pool_t *p
/* Create a cache with just one entry. */
SVN_ERR(svn_cache__create_inprocess(&cache,
- dup_revnum,
- APR_HASH_KEY_STRING,
- 1,
- 1,
- TRUE,
- pool));
+ serialize_revnum,
+ deserialize_revnum,
+ APR_HASH_KEY_STRING,
+ 1,
+ 1,
+ TRUE,
+ pool));
return basic_cache_test(cache, TRUE, pool);
}
Re: svn commit: r1072302 - in /subversion/trunk: ./ subversion/include/private/
subversion/libsvn_fs_fs/ subversion/libsvn_subr/ subversion/tests/libsvn_subr/
Posted by Stefan Fuhrmann <st...@alice-dsl.de>.
On 19.02.2011 12:12, Ivan Zhakov wrote:
> On Sat, Feb 19, 2011 at 13:47,<st...@apache.org> wrote:
>> Author: stefan2
>> Date: Sat Feb 19 10:47:16 2011
>> New Revision: 1072302
>>
>> URL: http://svn.apache.org/viewvc?rev=1072302&view=rev
>> Log:
>> Merge all changes (r1068724, r1068739) from the
>> integrate-cache-item-serialization branch.
>>
>> These patches introduce a very simple, limited purpose
>> serialization framework and use that to switch the
>> caching API from data copying to the more generally
>> applicable (de-)serialization mechanism.
>>
>> This makes all types of caches usable with all types
>> of cacheable items.
>>
> Hi Stefan,
>
> Why serialize/deserialize mechanism is better than data copying
> function?
This is not (de)serialization as in "write / parse some ASCII text"
but more basic as in "combine everything into a single, movable
binary chunk of data".
Basically, I simply concatenate all structs (memcpy) and replace
all pointers by local offsets. This requires some state management
during serialization but saves on allocations. This makes the
serialization part about as fast as standard copy code.
De-serialization is much faster, though: Only a single, aligned
allocation and memcpy followed by straightforward pointer
fix-up is necessary. The latter is equivalent to just setting the
pointer values during ordinary copying. Hence, we save greatly
on the allocation and copying part.
> For me implementing dup function is much easier and better
> for performance than serialize/deserialize. Is it possible to keep
> both mechanism in inprocess cache?
It is possible but not beneficial. We would need to keep code
for redundant functionality around that is generally slower and
limited in its applicability as it works for cache-inprocess only.
-- Stefan^2.
Re: svn commit: r1072302 - in /subversion/trunk: ./
subversion/include/private/ subversion/libsvn_fs_fs/ subversion/libsvn_subr/ subversion/tests/libsvn_subr/
Posted by Ivan Zhakov <iv...@visualsvn.com>.
On Sat, Feb 19, 2011 at 13:47, <st...@apache.org> wrote:
> Author: stefan2
> Date: Sat Feb 19 10:47:16 2011
> New Revision: 1072302
>
> URL: http://svn.apache.org/viewvc?rev=1072302&view=rev
> Log:
> Merge all changes (r1068724, r1068739) from the
> integrate-cache-item-serialization branch.
>
> These patches introduce a very simple, limited purpose
> serialization framework and use that to switch the
> caching API from data copying to the more generally
> applicable (de-)serialization mechanism.
>
> This makes all types of caches usable with all types
> of cacheable items.
>
Hi Stefan,
Why serialize/deserialize mechanism is better than data copying
function? For me implementing dup function is much easier and better
for performance than serialize/deserialize. Is it possible to keep
both mechanism in inprocess cache?
--
Ivan Zhakov