fix testing in next major release

[Benoit Chesneau <bc...@gmail.com>]

I would like to start to rewrite all of our tests to make them really relianle.

The problems that I want to fix are the following:

- try to remove most of the sleep w e have around
- remove JS tests when it's about testing the HTTP API. There are
actually no reason to test the HTTP API by faking a browser like we do
actually when we could test expected response against requests. The JS
tests should only be used eventually to test the the _change API or
CORS features, basically any browser features. But we are still doing
it wrong right now.
- Etap is not maintained anymore. I think we should move to eunit and
other standards things.


Thoughts?

- benoit

Re: fix testing in next major release

[Wendall Cada <we...@apache.org>]

I think this is a fantastic idea. I'd be interested in helping wherever 
necessary to make this happen.

Wendall

On 05/29/2013 06:35 AM, Benoit Chesneau wrote:
> I would like to start to rewrite all of our tests to make them really relianle.
>
> The problems that I want to fix are the following:
>
> - try to remove most of the sleep w e have around
> - remove JS tests when it's about testing the HTTP API. There are
> actually no reason to test the HTTP API by faking a browser like we do
> actually when we could test expected response against requests. The JS
> tests should only be used eventually to test the the _change API or
> CORS features, basically any browser features. But we are still doing
> it wrong right now.
> - Etap is not maintained anymore. I think we should move to eunit and
> other standards things.
>
>
> Thoughts?
>
> - benoit

Re: fix testing in next major release

[Robert Newson <rn...@apache.org>]

+1

On 29 May 2013 14:35, Benoit Chesneau <bc...@gmail.com> wrote:
> I would like to start to rewrite all of our tests to make them really relianle.
>
> The problems that I want to fix are the following:
>
> - try to remove most of the sleep w e have around
> - remove JS tests when it's about testing the HTTP API. There are
> actually no reason to test the HTTP API by faking a browser like we do
> actually when we could test expected response against requests. The JS
> tests should only be used eventually to test the the _change API or
> CORS features, basically any browser features. But we are still doing
> it wrong right now.
> - Etap is not maintained anymore. I think we should move to eunit and
> other standards things.
>
>
> Thoughts?
>
> - benoit

Re: fix testing in next major release

[Simon Metson <si...@cloudant.com>]

I'd be glad to help with this effort if it's the direction we choose to go in. 


On Monday, 3 June 2013 at 03:22, Dale Harvey wrote:

> I would be happy to come up with a hygenic example of what I think a common
> test suite could look like pretty soon

Re: fix testing in next major release

[Dale Harvey <da...@arandomurl.com>]

For Pouch we have spent a lot of time rewriting a lot of the couch tests to
be suitable for testing Pouch, we have put a pretty huge effort into
testing and although our current suite isnt a suitable replacement for the
couch suite, if it was possible I would definitely love to work on a single
suite that covered both projects.

For PouchDB we have to use async javascript, there isnt any budge for us on
that part.

Mocking xhr is messy, we just use a simple wrapper that uses http /
requests api in node.js and xhr in the browser, its proved very reliable,
for the last few months we have almost no false positives that werent
related to travis / saucelabs infrastructure.

This would require using the pouch interface to define the tests (the whole
point is to use a single interface to test multiple implementations) but
this is pretty much a direct (and better supported) replacement for CouchJS

I would be happy to come up with a hygenic example of what I think a common
test suite could look like pretty soon




On 2 June 2013 09:14, Russell Branca <ch...@gmail.com> wrote:

> +1
>
> -Russell
> On Jun 2, 2013 3:43 AM, "Noah Slater" <ns...@apache.org> wrote:
>
> > Big +1 from me.
> >
> > One of the ideas I had recently was to run each test 1,000 times (or
> > whatever) and calculate the probability of it failing. That should enable
> > us to prioritise the most brittle tests.
> >
> > But it seems like you have a much more radical idea in mind. +1 for being
> > radical.
> >
> >
> > On 29 May 2013 14:35, Benoit Chesneau <bc...@gmail.com> wrote:
> >
> > > I would like to start to rewrite all of our tests to make them really
> > > relianle.
> > >
> > > The problems that I want to fix are the following:
> > >
> > > - try to remove most of the sleep w e have around
> > > - remove JS tests when it's about testing the HTTP API. There are
> > > actually no reason to test the HTTP API by faking a browser like we do
> > > actually when we could test expected response against requests. The JS
> > > tests should only be used eventually to test the the _change API or
> > > CORS features, basically any browser features. But we are still doing
> > > it wrong right now.
> > > - Etap is not maintained anymore. I think we should move to eunit and
> > > other standards things.
> > >
> > >
> > > Thoughts?
> > >
> > > - benoit
> > >
> >
> >
> >
> > --
> > NS
> >
>

Re: fix testing in next major release

[Russell Branca <ch...@gmail.com>]

+1

-Russell
On Jun 2, 2013 3:43 AM, "Noah Slater" <ns...@apache.org> wrote:

> Big +1 from me.
>
> One of the ideas I had recently was to run each test 1,000 times (or
> whatever) and calculate the probability of it failing. That should enable
> us to prioritise the most brittle tests.
>
> But it seems like you have a much more radical idea in mind. +1 for being
> radical.
>
>
> On 29 May 2013 14:35, Benoit Chesneau <bc...@gmail.com> wrote:
>
> > I would like to start to rewrite all of our tests to make them really
> > relianle.
> >
> > The problems that I want to fix are the following:
> >
> > - try to remove most of the sleep w e have around
> > - remove JS tests when it's about testing the HTTP API. There are
> > actually no reason to test the HTTP API by faking a browser like we do
> > actually when we could test expected response against requests. The JS
> > tests should only be used eventually to test the the _change API or
> > CORS features, basically any browser features. But we are still doing
> > it wrong right now.
> > - Etap is not maintained anymore. I think we should move to eunit and
> > other standards things.
> >
> >
> > Thoughts?
> >
> > - benoit
> >
>
>
>
> --
> NS
>

Re: fix testing in next major release

[Noah Slater <ns...@apache.org>]

Big +1 from me.

One of the ideas I had recently was to run each test 1,000 times (or
whatever) and calculate the probability of it failing. That should enable
us to prioritise the most brittle tests.

But it seems like you have a much more radical idea in mind. +1 for being
radical.


On 29 May 2013 14:35, Benoit Chesneau <bc...@gmail.com> wrote:

> I would like to start to rewrite all of our tests to make them really
> relianle.
>
> The problems that I want to fix are the following:
>
> - try to remove most of the sleep w e have around
> - remove JS tests when it's about testing the HTTP API. There are
> actually no reason to test the HTTP API by faking a browser like we do
> actually when we could test expected response against requests. The JS
> tests should only be used eventually to test the the _change API or
> CORS features, basically any browser features. But we are still doing
> it wrong right now.
> - Etap is not maintained anymore. I think we should move to eunit and
> other standards things.
>
>
> Thoughts?
>
> - benoit
>



-- 
NS