You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@subversion.apache.org by gs...@apache.org on 2011/07/20 08:28:43 UTC
svn commit: r1148629 - in /subversion/trunk: notes/ra-serf-testing.txt
subversion/libsvn_ra_serf/util.c
Author: gstein
Date: Wed Jul 20 06:28:43 2011
New Revision: 1148629
URL: http://svn.apache.org/viewvc?rev=1148629&view=rev
Log:
Add an initial implementation of the testing framework for the new
pushback code.
Note: this doesn't work yet... but getting there.
Note: the pushback code (nor the testing) is still not active
* subversion/libsvn_ra_serf/util.c:
(PBTEST_ACTIVE: new control symbol
(pbtest_step): current step of the testing sequence
(PBTEST_STATE): helper macro to determine the current PENDING state,
according to ra-serf-testing.txt
(pbtest_desc_t): new descriptor for each test step
(pbtest_description): the set of descriptors for all steps
(PBTEST_SET_PAUSED): new helper to update the ->paused flag
(PBTEST_MAYBE_STEP): helper macro for invoking maybe_next_step()
(PBTEST_FORCE_SPILL): new macro for determine whether to force content
to be spilled into a file
(maybe_next_step): defined when PBTEST_ACTIVE. possibly moves the step
forward. plenty of good debug output.
(write_to_pending): possibly force a spill to the file, or maybe deny
spillage.
(inject_to_parser): potentially ignore the callback's assignment of the
PAUSED flag, setting it appropriately for the current step.
(svn_ra_serf__process_pending): possibly update the PAUSED state, a
little debug output, and manage some of the forced content injection.
(svn_ra_serf__handle_xml_parser): advance the step on the initial
invocation. advance after each storage into PENDING. make sure the
PAUSED flag is correct before testing it.
* notes/ra-serf-testing.txt:
(...): updates
Modified:
subversion/trunk/notes/ra-serf-testing.txt
subversion/trunk/subversion/libsvn_ra_serf/util.c
Modified: subversion/trunk/notes/ra-serf-testing.txt
URL: http://svn.apache.org/viewvc/subversion/trunk/notes/ra-serf-testing.txt?rev=1148629&r1=1148628&r2=1148629&view=diff
==============================================================================
--- subversion/trunk/notes/ra-serf-testing.txt (original)
+++ subversion/trunk/notes/ra-serf-testing.txt Wed Jul 20 06:28:43 2011
@@ -281,24 +281,44 @@ possible.
9a. network: unpaused. extra content is TEST 1.4.
9b. loop: unpaused. inject all memory content. TEST 3.4. step on state (6)
10. paused. content arrival is TEST 1.6.
-11a. network: unpaused. content arrival is TEST 2.6. step on state (6)
+11a. network: unpaused. content arrival is TEST 2.6.
11b. loop: no injection
12a. network: unpaused. extra content is TEST 2.6.
12b. loop: unpaused. inject all disk content. TEST 3.6. step on state (7)
-13. paused. content arrival is TEST 1.7
+13. paused. content arrival is TEST 1.7. step on state (6)
14. return to default/normal processing
note: "extra content" means that TEST has been previously tested, so
we don't really need this particular test. but since the network
content is uncontrolled, we may end up (re)testing.
+note: we advance the step when reaching a particular state. if the
+ state is not mentioned, then we advance once new content has
+ been received (which actually means the same state as before),
+ or we advance after injection to the parser.
+note: we cannot detect whether content exists in the spill file due to
+ the information we (currently) record. thus, we have to do the
+ transition manually:
+ step 6: writing the spill moves us to state (4)
+ step 9b: injecting all moves us to state (6)
+ step 12b: injecting all moves us to state (7)
+ step 13: appending to spill moves us to state (6)
+note: after each injection of pending content into the parser, we
+ will pause the parser. that will immediately cause the
+ processing of the PENDING data to stop.
IMPLEMENTATION NOTES
-* gotta figure out low-impact
+* gotta figure out low-impact (this can all live in util.c)
* array of structures with: paused, inject, force_spill
-* set ->paused on each step increment, since we don't know if the next
+* set ->paused on each step increment since we don't know if the next
entry point will be the network or the processing loop
+* set ->paused on each return from the xml parser callbacks so that we
+ do not have to invade update.c (which wouldn't know when we turn off
+ the debugging at step 14)
+* the processing loop may clear ->paused and call the pending
+ processing function, which can then reset ->paused and exit if
+ necessary for the current step
* there are three content injection steps. transition to next step is
straight-forward after injection is complete
* transition based on network has three transitions from the unpaused
@@ -310,29 +330,5 @@ IMPLEMENTATION NOTES
independent of the memory_size. since we will put further pending
data into the spill file, we don't need to check force_spill ever
again. oh... we can just look for step==6 rather than a flag.
+* transitions occur on five of the seven states (not 1 or 5)
-
-
-------------
-OLD NOTES
-
-These combine into the following test scenarios to exercise the two
-actions against each of the ten data states.
-
- #### redo this list
-
-
- 1) REPORT completes with no pausing
- 2) REPORT requires in-memory (only) buffering:
- a) buffer empties while network content arrives again
- b) network content completes while buffer has content
- c) buffer empties, then later needs buffering again
- d) partial in-mem consumption, new content lands into membuf
- 3) REPORT requires on-disk buffering:
- a) diskbuf empties while network content arrives again
- b) network content completes while diskbuf has content
- c) diskbuf empties, then more buffering is needed again
- (note: due to the code, this will go onto the disk rather
- than back to the in-mem buffer)
- d) partial in-mem consumption, new content lands in diskbuf
- e) partial diskbuf consumption, new content lands in diskbuf
Modified: subversion/trunk/subversion/libsvn_ra_serf/util.c
URL: http://svn.apache.org/viewvc/subversion/trunk/subversion/libsvn_ra_serf/util.c?rev=1148629&r1=1148628&r2=1148629&view=diff
==============================================================================
--- subversion/trunk/subversion/libsvn_ra_serf/util.c (original)
+++ subversion/trunk/subversion/libsvn_ra_serf/util.c Wed Jul 20 06:28:43 2011
@@ -103,7 +103,65 @@ struct svn_ra_serf__pending_t {
#define SPILL_SIZE 1000000
/* See notes/ra-serf-testing.txt for some information on testing this
- new "paused" feature. */
+ new "paused" feature (aka network pushback).
+
+ Define PBTEST_ACTIVE, if you would like to run the pushback tests. */
+#undef PBTEST_ACTIVE
+#ifdef PBTEST_ACTIVE
+
+static int pbtest_step = 0;
+
+/* Note: this cannot resolve states 5 and 7. */
+#define PBTEST_STATE(p) \
+ ((p) == NULL ? 1 \
+ : ((p)->spill == NULL ? ((p)->head == NULL ? 2 : 3) \
+ : ((p)->head == NULL ? 6 : 4)))
+
+typedef struct {
+ svn_boolean_t paused; /* pause the parser on this step? */
+ svn_boolean_t inject; /* inject pending content on this step? */
+ int when_next; /* when to move to the next step? */
+ const char *completed; /* what test was completed? */
+} pbtest_desc_t;
+
+static const pbtest_desc_t pbtest_description[] = {
+ { 0 }, /* unused */
+ { TRUE, FALSE, 3, "1.1" },
+ { TRUE, FALSE, 3, "1.3" },
+ { FALSE, FALSE, 3, "2.3" },
+ { FALSE, TRUE, 2, "3.3" }, /* WHEN_NEXT is ignored due to INJECT */
+ { TRUE, FALSE, 3, "1.2" },
+ { TRUE, FALSE, 4, NULL },
+ { TRUE, FALSE, 4, "1.4" },
+ { FALSE, FALSE, 4, "2.4" },
+ { FALSE, TRUE, 6, "3.4" }, /* WHEN_NEXT is ignored due to INJECT */
+ { TRUE, FALSE, 6, "1.6" },
+ { FALSE, FALSE, 6, "2.6" },
+ { FALSE, TRUE, 7, "3.6" }, /* WHEN_NEXT is ignored due to INJECT */
+ { TRUE, FALSE, 6, "1.7" },
+ { 0 } /* unused */
+};
+
+#define PBTEST_SET_PAUSED(ctx) \
+ (pbtest_step < 14 \
+ ? (ctx)->paused = pbtest_description[pbtest_step].paused \
+ : FALSE)
+
+#define PBTEST_MAYBE_STEP(ctx) maybe_next_step(ctx)
+
+#define PBTEST_FORCE_SPILL() (pbtest_step == 6)
+
+#else /* PBTEST_ACTIVE */
+
+/* Be wary with the definition of these macros so that we don't
+ end up with "statement with no effect" warnings. Obviously, this
+ depends upon particular usage, which is easy to verify. */
+#define PBTEST_SET_PAUSED(ctx) /* empty */
+#define PBTEST_MAYBE_STEP(ctx) /* empty */
+
+#define PBTEST_FORCE_SPILL() FALSE
+
+#endif /* PBTEST_ACTIVE */
@@ -1250,6 +1308,48 @@ add_done_item(svn_ra_serf__xml_parser_t
}
+#ifdef PBTEST_ACTIVE
+
+/* Determine whether we should move to the next step. Print out the
+ transition for debugging purposes. */
+static void
+maybe_next_step(svn_ra_serf__xml_parser_t *parser)
+{
+ const pbtest_desc_t *desc;
+ int state;
+
+ /* This would fail the state transition, but for clarity... just return
+ when the testing has completed. */
+ if (pbtest_step == 14)
+ return;
+
+ desc = &pbtest_description[pbtest_step];
+ state = PBTEST_STATE(parser->pending);
+
+ if (desc->inject || state == desc->when_next || pbtest_step == 0)
+ {
+ ++pbtest_step;
+
+ if (desc->completed != NULL)
+ SVN_DBG(("PBTEST: completed TEST %s\n", desc->completed));
+
+ /* Pause the parser based on the new step's config. */
+ ++desc;
+ parser->paused = desc->paused;
+
+ SVN_DBG(("PBTEST: advanced: step=%d paused=%d inject=%d\n",
+ pbtest_step, desc->paused, desc->inject));
+ }
+ else
+ {
+ SVN_DBG(("PBTEST: step[%d]: state=%d waiting_for=%d\n",
+ pbtest_step, state, desc->when_next));
+ }
+}
+
+#endif /* PBTEST_ACTIVE */
+
+
/* Get a buffer from the parsing context. It will come from the free list,
or allocated as necessary. */
static struct pending_buffer_t *
@@ -1293,10 +1393,18 @@ write_to_pending(svn_ra_serf__xml_parser
/* We do not (yet) have a spill file, but the amount stored in memory
has grown too large. Create the file and place the pending data into
- the temporary file. */
- if (ctx->pending->spill == NULL
- && ctx->pending->memory_size > SPILL_SIZE)
- {
+ the temporary file.
+
+ For testing purposes, there are points when we may want to
+ create the spill file, regardless. */
+ if (PBTEST_FORCE_SPILL()
+ || (ctx->pending->spill == NULL
+ && ctx->pending->memory_size > SPILL_SIZE))
+ {
+#ifdef PBTEST_ACTIVE
+ /* Only allow a spill file for steps 6 or later. */
+ if (pbtest_step >= 6)
+#endif
SVN_ERR(svn_io_open_unique_file3(&ctx->pending->spill,
NULL /* temp_path */,
NULL /* dirpath */,
@@ -1371,6 +1479,10 @@ inject_to_parser(svn_ra_serf__xml_parser
if (ctx->error && !ctx->ignore_errors)
return svn_error_trace(ctx->error);
+ /* We may want to ignore the callbacks choice for the PAUSED flag.
+ Set this value, as appropriate. */
+ PBTEST_SET_PAUSED(ctx);
+
return SVN_NO_ERROR;
}
@@ -1383,6 +1495,13 @@ svn_ra_serf__process_pending(svn_ra_serf
svn_error_t *err;
apr_off_t output_unused;
+ /* We may need to repair the PAUSED state when testing. */
+ PBTEST_SET_PAUSED(parser);
+
+#ifdef PBTEST_ACTIVE
+ SVN_DBG(("process: paused=%d\n", parser->paused));
+#endif
+
/* Fast path exit: already paused, or nothing to do. */
if (parser->paused || parser->pending == NULL)
return SVN_NO_ERROR;
@@ -1420,6 +1539,17 @@ svn_ra_serf__process_pending(svn_ra_serf
return SVN_NO_ERROR;
}
+#ifdef PBTEST_ACTIVE
+ /* For steps 4 and 9, we wait until all of the memory content has been
+ injected. At that point, we can take another step which will pause
+ the parser, and we'll need to exit. */
+ if (pbtest_step == 4 || pbtest_step == 9)
+ {
+ PBTEST_MAYBE_STEP(parser);
+ return SVN_NO_ERROR;
+ }
+#endif
+
/* If we don't have a spill file, then we've exhausted all
pending content. */
if (parser->pending->spill == NULL)
@@ -1481,6 +1611,11 @@ svn_ra_serf__process_pending(svn_ra_serf
XML_ParserFree(parser->xmlp);
add_done_item(parser);
}
+
+ /* For testing step 12, we have written all of the disk content. This
+ will advance to step 13 and pause the parser again. */
+ PBTEST_MAYBE_STEP(parser);
+
return SVN_NO_ERROR;
}
@@ -1538,6 +1673,10 @@ svn_ra_serf__handle_xml_parser(serf_requ
{
XML_SetCharacterDataHandler(ctx->xmlp, cdata_xml);
}
+
+ /* This is the first invocation. Possibly move to step 1 of the
+ testing sequence. */
+ PBTEST_MAYBE_STEP(ctx);
}
/* If we are storing content into a spill file, then move to the end of
@@ -1559,11 +1698,19 @@ svn_ra_serf__handle_xml_parser(serf_requ
status = serf_bucket_read(response, PARSE_CHUNK_SIZE, &data, &len);
+#ifdef PBTEST_ACTIVE
+ SVN_DBG(("response: len=%d\n", (int)len));
+#endif
+
if (SERF_BUCKET_READ_ERROR(status))
{
return svn_error_wrap_apr(status, NULL);
}
+ /* Ensure that the parser's PAUSED state is correct before we test
+ the flag. */
+ PBTEST_SET_PAUSED(ctx);
+
/* Note: once the callbacks invoked by inject_to_parser() sets the
PAUSED flag, then it will not be cleared. write_to_pending() will
only save the content. Logic outside of serf_context_run() will
@@ -1577,6 +1724,13 @@ svn_ra_serf__handle_xml_parser(serf_requ
if (ctx->paused || HAS_PENDING_DATA(ctx->pending))
{
err = write_to_pending(ctx, data, len, pool);
+
+ /* We may have a transition to a next step.
+
+ Note: this only happens on writing to PENDING. If the
+ parser is unpaused, then we will never change state within
+ this network-reading loop. */
+ PBTEST_MAYBE_STEP(ctx);
}
else
#endif