Test case confusion.

[Mo DeJong <su...@bayarea.net>]

Hi all.

I have started to poke around in the test suite again and I am running into
some things that just don't make sense to me. I am hoping the person
that designed these tests can shed some light on a couple of things.

For one thing, why is there this sandbox() method in each .py test file?

def sandbox(x):
  return "basic_tests-" + `test_list.index(x)`

A call like:

sbox = sandbox(basic_status)

Will return a string like "basic_tests-0" which will be used to create a unique name
for the repo and checkout dirs for the test case. Why would we want to do that instead
of just assuming that each test function has a globally unique name and using the test name
as the dir name?

The current implementation makes it hard to map from the directory names in
subversion/tests/clients/cmdline/repositories to the actual test cases. You need
to manually count the number of entries in the test_list to find the right one.

Instead of:

  sbox = sandbox(basic_commit)

We could just use:

  sbox = "basic_status"

That would make it really easy to figure out which test a given repo and
WC belonged to.

The other problem I am having has to do with running a single test from the
make check command line. The make check rule start off like so:

check: $(TEST_DEPS) @FS_TEST_DEPS@
	@logfile=`pwd`/tests.log ; \
	echo > $$logfile ; \
	failed=no ; \
	list='$(TEST_PROGRAMS) @FS_TEST_PROGRAMS@'; for prog in $$list; do \

So, I can run a single test case from the command line by doing this:

make check TEST_PROGRAMS="" \
FS_TEST_PROGRAMS="subversion/tests/clients/cmdline/basic_tests.py"

As you can see, that is quite a bother. What if we made a small change
to the Makefile so that is easier?

% make check TESTS="subversion/tests/clients/cmdline/basic_tests.py"

It could be implemented something like this (not a real patch)

Index: Makefile.in
-----------------------
+# Users can run 'make check TESTS=...' from the command line to run
+# a single test or set of tests
+TESTS = $(TEST_PROGRAMS) @FS_TEST_PROGRAMS@
+
 check: $(TEST_DEPS) @FS_TEST_DEPS@
        @logfile=`pwd`/tests.log ; \
        echo > $$logfile ; \
        failed=no ; \
-       list='$(TEST_PROGRAMS) @FS_TEST_PROGRAMS@'; for prog in $$list; do \
+       list='$(TESTS)'; for prog in $$list; do \

cheers
Mo

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org

Re: Test case confusion.

[cm...@collab.net]

Mo DeJong <su...@bayarea.net> writes:

> A call like:
> 
> sbox = sandbox(basic_status)
> 
> Will return a string like "basic_tests-0" which will be used to
> create a unique name for the repo and checkout dirs for the test
> case. Why would we want to do that instead of just assuming that
> each test function has a globally unique name and using the test
> name as the dir name?
> 
> The current implementation makes it hard to map from the directory
> names in subversion/tests/clients/cmdline/repositories to the actual
> test cases. You need to manually count the number of entries in the
> test_list to find the right one.

Nah, just run './basic_tests.py list' and see which function
description matches the one that concerns you.

> The other problem I am having has to do with running a single test from the
> make check command line. The make check rule start off like so:
> 
> check: $(TEST_DEPS) @FS_TEST_DEPS@
> 	@logfile=`pwd`/tests.log ; \
> 	echo > $$logfile ; \
> 	failed=no ; \
> 	list='$(TEST_PROGRAMS) @FS_TEST_PROGRAMS@'; for prog in $$list; do \
> 
> So, I can run a single test case from the command line by doing this:
> 
> make check TEST_PROGRAMS="" \
> FS_TEST_PROGRAMS="subversion/tests/clients/cmdline/basic_tests.py"
> 
> As you can see, that is quite a bother. What if we made a small change
> to the Makefile so that is easier?
> 
> % make check TESTS="subversion/tests/clients/cmdline/basic_tests.py"
> 
> It could be implemented something like this (not a real patch)
> 
> Index: Makefile.in
> -----------------------
> +# Users can run 'make check TESTS=...' from the command line to run
> +# a single test or set of tests
> +TESTS = $(TEST_PROGRAMS) @FS_TEST_PROGRAMS@
> +
>  check: $(TEST_DEPS) @FS_TEST_DEPS@
>         @logfile=`pwd`/tests.log ; \
>         echo > $$logfile ; \
>         failed=no ; \
> -       list='$(TEST_PROGRAMS) @FS_TEST_PROGRAMS@'; for prog in $$list; do \
> +       list='$(TESTS)'; for prog in $$list; do \

This all seems reasonably.  Makes sense to have single variable
containing all the tests so long as we don't do anything different in
TEST_DEPS vs. FS_TEST_DEPS.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@subversion.tigris.org
For additional commands, e-mail: dev-help@subversion.tigris.org