You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by Jeff Hammerbacher <ha...@cloudera.com> on 2009/11/19 11:14:47 UTC
Shared schemas for testing?
Hey,
We've got a test/schemata/ directory, but the only schema is the
interop.avsc (everything else is a protocol). Meanwhile, we spend a lot of
lines of code building big strings in our testing code to probe various
aspects of schema parsing (e.g.
src/test/java/org/apache/avro/TestSchema.java, src/test/py/testio.py). Would
it make sense to extract these schemas into files under test/schemata/ and
pull them in for the tests in each language? By doing so, we could also
insure that the various language bindings are being tested in a consistent
fashion.
Let me know what you think.
Thanks,
Jeff
Re: Shared schemas for testing?
Posted by Doug Cutting <cu...@apache.org>.
Matt Massie wrote:
> Alternatively, we could use the following layout (which I *think* I like
> better)...
>
> tests/
> invalid_schemas/
> broken.json
> wrong.json
> valid_data/
> foo_test/
> schema.json
> json_data/
I like this better. It's a flatter structure and doesn't try to be
overly general.
Doug
Re: Shared schemas for testing?
Posted by Matt Massie <ma...@cloudera.com>.
I should have known better than to send a sentence that starts with "of
course"... :)
We could support invalid data tests by having two top-level test trees. One
for schema tests and one for data tests.
tests/
schema_tests/
pass/
valid_schema.json
good_schema.json
fail/
bad_schema_1.json
broken_schema_2.json
out_in_left_field_schema.json
data_tests/
pass/
foo_test/
schema.json
json_data/
valid_json_data_for_schema.json
more_valid_json_data_matching_test_schema.json
binary_data/
valid_binary_data.bin
good_binary_data.bin
bar_test/
...
fail/
baz_test/
schema.json
json_data/
corrupt_invalid_data.json
simply_wrong_data.json
binary_data/
more_bad_data.bin
epic_fail_test/
schema.json
...
This layout allows for schema only tests and invalid data tests. It an
error for any schemas in the data "data_tests" directory to be invalid. All
schema in the data test directory are valid... it's the data that is wrong.
Alternatively, we could use the following layout (which I *think* I like
better)...
tests/
invalid_schemas/
broken.json
wrong.json
valid_data/
foo_test/
schema.json
json_data/
...
binary_data/
...
bar_test/
schema.json
json_data/
...
invalid_data/
baz_test/
schema.json
json_data/
...
binary_data/
...
The "json_data" and "binary_data" directories are optional so if you just
want to test the schema you don't create those subdirectories in the test.
As with the first layout, it is invalid to have a bad schema under the
"valid_data" or "invalid_data" directories (because, of course, we need a
valid schema to see if the data is (in)valid).
-Matt
On Thu, Nov 19, 2009 at 11:04 AM, Doug Cutting <cu...@apache.org> wrote:
> Matt Massie wrote:
>
>> Of course, there would be no data tests for invalid schemas.
>>
>
> Although we probably do want some invalid data, to check that errors are
> caught.
>
> Doug
>
Re: Shared schemas for testing?
Posted by Doug Cutting <cu...@apache.org>.
Matt Massie wrote:
> Of course, there would be no data tests for invalid schemas.
Although we probably do want some invalid data, to check that errors are
caught.
Doug
Re: Shared schemas for testing?
Posted by Matt Massie <ma...@cloudera.com>.
On Thu, Nov 19, 2009 at 10:28 AM, Doug Cutting <cu...@apache.org> wrote:
> Matt Massie wrote:
>
>> I'd recommend we consider using the
>> approach more generally.
>>
>
> For stuff besides parsing it's be more complicated to share test data.
>
> Perhaps we could have three files per schema: the schema, a data file
> containing binary instances of that schema and a data file containing json
> instances of that schema. The generic tester could read the data files
> check that their instances match.
>
The main thing we need is a standardized directory structure. For
example...
tests/
pass/
test_a/
schema.json
json_data/
one_test_data.json
two_test_data.json
etc_test_data.json
avro_data/
binary_test_data.bin
another_binary_test_data_file.bin
test_b/
...
fail/
Of course, there would be no data tests for invalid schemas. Once we agree
on a directory structure, we can all write our code once and then focus on
populating our test tree.
> Or we could standardize on a simple random datum generator. Then we could
> have, for each schema, random seeds paired with binary instances. Then one
> can generate datastructure instances given the seed and check that they're
> equal to the instance read from the test file, etc. This would be a
> stronger test.
>
Once we agree on the directory structure, we can have all the Avro
implementations generate test data any number of different ways. +1 on the
random seed idea.
-Matt
Re: Shared schemas for testing?
Posted by Doug Cutting <cu...@apache.org>.
Matt Massie wrote:
> I'd recommend we consider using the
> approach more generally.
For stuff besides parsing it's be more complicated to share test data.
Perhaps we could have three files per schema: the schema, a data file
containing binary instances of that schema and a data file containing
json instances of that schema. The generic tester could read the data
files check that their instances match.
Or we could standardize on a simple random datum generator. Then we
could have, for each schema, random seeds paired with binary instances.
Then one can generate datastructure instances given the seed and check
that they're equal to the instance read from the test file, etc. This
would be a stronger test.
Doug
Re: Shared schemas for testing?
Posted by Matt Massie <ma...@cloudera.com>.
In my C tests, I have "pass" and "fail' directories inside my "schema_tests"
directory. This approach allows the schema test to easily be extended
without modifying any code (for example, yesterday Eli dropped in some RPC
schema tests into "pass" directory and re-ran the tests which automatically
pulled in the new schemas to test). I'd recommend we consider using the
approach more generally.
I'm +1 on using the less pedantic "schemas" over "schemata".
-Matt
On Thu, Nov 19, 2009 at 9:41 AM, Doug Cutting <cu...@apache.org> wrote:
> Jeff Hammerbacher wrote:
>
>> Would it make sense to extract these schemas into files under
>> test/schemata/ and
>> pull them in for the tests in each language?
>>
>
> Yes, that makes sense. Also, Matt's C json test suite could probably be
> shared with C++, since both implement json parsers.
>
> Note that after AVRO-163 these will go in src/share. Elsewhere in the
> project (e.g., in the spec) we avoid the pedantic "schemata". Should we
> avoid it here, naming this src/share/schemas, or stick with it?
>
> Doug
>
Re: Shared schemas for testing?
Posted by Doug Cutting <cu...@apache.org>.
Jeff Hammerbacher wrote:
> Would it make sense to extract these schemas into files under test/schemata/ and
> pull them in for the tests in each language?
Yes, that makes sense. Also, Matt's C json test suite could probably be
shared with C++, since both implement json parsers.
Note that after AVRO-163 these will go in src/share. Elsewhere in the
project (e.g., in the spec) we avoid the pedantic "schemata". Should we
avoid it here, naming this src/share/schemas, or stick with it?
Doug
Re: Shared schemas for testing?
Posted by Patrick Hunt <ph...@apache.org>.
Sounds good to me, having more interoperability testing is a must IMO.
To that end I've been fooling with an avro schema fuzzer which you can
glance at here: http://bit.ly/cDHST
It's no where near ready yet, in particular I'm currently fuzzing at the
protocol level and I should probably be doing things at the schema level
(is there a way to serialize RPC messages directly to disk, ala thrift,
would that be difficult?) The current implementation works by generating
a schema, then parsing that schema in both python (directly) and java
(via call to specificcompiler and then javac).
Patrick
Jeff Hammerbacher wrote:
> Hey,
>
> We've got a test/schemata/ directory, but the only schema is the
> interop.avsc (everything else is a protocol). Meanwhile, we spend a lot of
> lines of code building big strings in our testing code to probe various
> aspects of schema parsing (e.g.
> src/test/java/org/apache/avro/TestSchema.java, src/test/py/testio.py). Would
> it make sense to extract these schemas into files under test/schemata/ and
> pull them in for the tests in each language? By doing so, we could also
> insure that the various language bindings are being tested in a consistent
> fashion.
>
> Let me know what you think.
>
> Thanks,
> Jeff
>