tests for doc examples?

[Myrna van Lunteren <m....@gmail.com>]

Hi,

I've been wondering and worrying about our doc examples.

Before contribution to apache, we did a couple of passes to test and verify
all doc examples. But this is a very time-consuming and error-prone process
for various reasons: easy for an individual to miss a snippet, snippets are
not written with tests in mind and don't easily follow on each other and so
every snippet needs to have a separate test written...or get tested
manually...
Also examples continue to get changed and for new code we should get new
examples, and we now have a demo database (toursdb) that we probably should
try to use...And I don't see that anyone has time available to test all the
snippets.

I think we should have automated tests for the doc examples.

Or at least as many as possible.
I am thinking, it should be possible to extract the example code from the
dita files and automatically build tests based on it.
We could mark the snippets that are easy to be wrapped into an SQL examples
one way, java snippets another, full java classes yet another, extract them,
and refresh the tests that way...(nightly? monthly? at code freeze dates?)

Maybe add a specific string to each example so we can search the dita files
for that. Or maybe there already is a specific sort of property/tag for code
snippets?

This also has implications for the examples - they should be (re)written
with wrapping into a test in mind.

Does this seem doable/feasible/sensible?

Myrna

Re: tests for doc examples?

[Jeff Levitt <de...@mylevita.com>]

--- Rajesh Kartha <ka...@gmail.com> wrote:

> Myrna van Lunteren wrote:
> 
> > Hi,
> >  
> > I've been wondering and worrying about our doc
> examples.
> >  
> > Before contribution to apache, we did a couple of
> passes to test and 
> > verify all doc examples. But this is a very
> time-consuming and 
> > error-prone process for various reasons: easy for
> an individual to 
> > miss a snippet, snippets are not written with
> tests in mind and don't 
> > easily follow on each other and so every snippet
> needs to have a 
> > separate test written...or get tested manually...
> > Also examples continue to get changed and for new
> code we should get 
> > new examples, and we now have a demo database
> (toursdb) that we 
> > probably should try to use...And I don't see that
> anyone has time 
> > available to test all the snippets.
> >  
> > I think we should have automated tests for the doc
> examples.
> >  
> > Or at least as many as possible.
> > I am thinking, it should be possible to extract
> the example code from 
> > the dita files and automatically build tests based
> on it.
> > We could mark the snippets that are easy to be
> wrapped into an SQL 
> > examples one way, java snippets another, full java
> classes yet 
> > another, extract them, and refresh the tests that
> way...(nightly? 
> > monthly? at code freeze dates?)
> >  
> > Maybe add a specific string to each example so we
> can search the dita 
> > files for that. Or maybe there already is a
> specific sort of 
> > property/tag for code snippets?
> >  
> > This also has implications for the examples - they
> should be 
> > (re)written with wrapping into a test in mind.
> >  
> > Does this seem doable/feasible/sensible?
> >  
> > Myrna
> 
> +1. Would be really useful.
> 
> With the toursdb demo availabe in the codeline, I
> think all the queries 
> should work as is. A great deal of attention needs
> to be paid
> to the results returned by the sample queries. I
> noticed, for ex. 
> DERBY-994, LEFT/RIGHT OUTER JOIN examples, the
> queries worked just fine 
> - but
> the results and description did not make sense at
> all.
> 
> 
> -Rajesh
> 
> 

I think it would be useful.  I'm pretty sure if you
searched the DITA for the content inside <codeblock>
tags, you'de have the examples.  But you'd need some
way of differentiating snippets, full code, etc.  Wht
would need to be done is:

A.  Define the kinds of examples (snippet, full code,
etc.)
B.  Do a one-time search of the <codeblocks>, and
determine what type each example is.
C.  Put something in the codeblock tag, whether it is
an attribute on the codeblock tag itself (for example:
<codeblock type="snippet">) or by leading the example
with a comment (for example:  

-- Example: SNIPPET title="CREATE TABLE statement
example"


I'll look into the attribute method.  Great idea!

Re: tests for doc examples?

[Rajesh Kartha <ka...@gmail.com>]

Myrna van Lunteren wrote:

> Hi,
>  
> I've been wondering and worrying about our doc examples.
>  
> Before contribution to apache, we did a couple of passes to test and 
> verify all doc examples. But this is a very time-consuming and 
> error-prone process for various reasons: easy for an individual to 
> miss a snippet, snippets are not written with tests in mind and don't 
> easily follow on each other and so every snippet needs to have a 
> separate test written...or get tested manually...
> Also examples continue to get changed and for new code we should get 
> new examples, and we now have a demo database (toursdb) that we 
> probably should try to use...And I don't see that anyone has time 
> available to test all the snippets.
>  
> I think we should have automated tests for the doc examples.
>  
> Or at least as many as possible.
> I am thinking, it should be possible to extract the example code from 
> the dita files and automatically build tests based on it.
> We could mark the snippets that are easy to be wrapped into an SQL 
> examples one way, java snippets another, full java classes yet 
> another, extract them, and refresh the tests that way...(nightly? 
> monthly? at code freeze dates?)
>  
> Maybe add a specific string to each example so we can search the dita 
> files for that. Or maybe there already is a specific sort of 
> property/tag for code snippets?
>  
> This also has implications for the examples - they should be 
> (re)written with wrapping into a test in mind.
>  
> Does this seem doable/feasible/sensible?
>  
> Myrna

+1. Would be really useful.

With the toursdb demo availabe in the codeline, I think all the queries 
should work as is. A great deal of attention needs to be paid
to the results returned by the sample queries. I noticed, for ex. 
DERBY-994, LEFT/RIGHT OUTER JOIN examples, the queries worked just fine 
- but
the results and description did not make sense at all.


-Rajesh