futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Sam Bisbee <sa...@sbisbee.com>]

On Wed, Oct 19, 2011 at 3:52 PM, Jan Lehnardt <ja...@apache.org> wrote:
>
> On Oct 19, 2011, at 21:25 , Sam Bisbee wrote:
>
>> Yeah, it looks like it's just an unstable test. Which makes me wonder
>> why we even have it or include it in the release process.
>
> It is only unstable in certain environments and we are constantly improving all tests to be more stable in all environments, but the browser world is harsh :)

I'm all for improvement. :) But is this improvement being tracked
anywhere, JIRA or otherwise? And if not, then could it be? I'm sure
that folks (self included) would be willing to pitch in if we knew
what had to be done.

> I'd still consider having the test better than not having it at all :)

I semi remember this being discussed before, but are the Futon tests
meant to test CouchDB, Futon, or the integration of the two?

Cheers,

--
Sam Bisbee

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Paul Davis <pa...@gmail.com>]

On Wed, Oct 19, 2011 at 10:38 PM, J. Lee Coltrane
<le...@projectmastermind.com> wrote:
>
>> For what it's worth, a CLI based test system is what I was imagining
>> as well. Take Futon out of the mix and test CouchDB.
>
> IMO, If CouchDB is intended to be a server that can be accessed from
> the browser directly, then there should continue to be some kind of
> browser-based test suite that would serve to confirm this capability.
>

CouchDB is intended to be a server that can be accessed from HTTP
clients. Browsers are but one of a huge range of clients. I do agree
that there should be a browser based test suite, but I'm proposing
that these browser based tests should be testing the browser and not
testing CouchDB internals.

>
> I have been looking closely at the Futon tests in 1.1.0 for the last
> several days, with the idea that I might begin to clean them up a bit
> as time permits.
>
> I have found that, while some of these test failures are totally bogus,
> *some* of them actually do stem from real issues -- minor
> incompatibilities between CouchDB's http interface, and the internal
> mechanisms of modern browsers (XHR, caching, etc).
>
> These are problems that we're not going to catch with a stateless,
> cache-less http client running on the CLI.  (I can provide examples)
>

And I can provide examples where having a stateful caching client
merely exists to confound the test code. The issue is that using a
stateful caching client means that all tests have to account for this,
even tests that have nothing to do with such things.

> These issues have the potential to cause real problems for
> developers of real browser-based apps "in the wild".  That means,
> there's valuable info to be gathered from the browser tests, Iff we
> can clean them up, and make them behave consistently; so that
> when they fail or succeed, we can actually trust the results.
>
>
> After digging around a good bit, I can see no reason why the existing
> tests couldn't be cleaned up and made to work correctly in all current
> versions of major browsers.  I also see no reason why the same tests
> couldn't be used successfully from the CLI and `make check` as well.
>

There are some tests that make use of XHR directly which are incapable
of being run from the CLI test runner for one. There are also issues
in differences of caching implementations. There are even differences
of caching against localhost vs a remote server. I consider every time
I've had to diagnose a difference in browser behavior as an example of
precisely why (most of) these tests do not belong in a browser. Not
only is this a waste of time, it merely serves to make the test suite
less trustable when an error occurs.

We can hand wave about cleaning up the tests to make them more
reliable, but that's ignoring the fact that we're running the test
suite in huge monolithic environments that have a decades long history
of maddeningly subtle different semantics.

> I do see significant benefits to using the same javascript test code in
> all environments we test.
>

What's the benefit to maintaing assertions in the browser about view
output according to the UCA. Or whether or not the database is leaking
file descriptors. Or couch_os_process is properly caching couchjs
processes?

> -Lee
> (irc: coltr)
>

I don't mean to be cranky at you directly, but I am quite tired of
dealing with browsers to test a database. It was a bad idea to start
with and I've been trying to argue for the change for years now.

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Paul Davis <pa...@gmail.com>]

Honestly, I was looking for the video of the one lady screaming to
provide commentary on they "I agree with what he said" comment. But I
failed slightly, but only slightly enough to illicit an awesome
introspection on a random Beatles video.

On Thu, Oct 20, 2011 at 11:17 PM, Noah Slater <ns...@tumbolia.org> wrote:
> Wait, did you post that because me and you rock this project like the
> Beatles rocked America? Or did you post it with the intention of the song
> She Loves You being a sort of meta-commentary on our enviable, and now
> infamous, bromance?
>
> On Fri, Oct 21, 2011 at 5:13 AM, Paul Davis <pa...@gmail.com>wrote:
>
>> http://www.youtube.com/watch?feature=player_detailpage&v=G6j5bve7O5E#t=109s
>>
>> On Thu, Oct 20, 2011 at 7:08 PM, Noah Slater <ns...@tumbolia.org> wrote:
>> > +1 on all the stuff Paul said.
>> >
>> > On Thu, Oct 20, 2011 at 9:25 PM, Robert Newson <rn...@apache.org>
>> wrote:
>> >
>> >> I'll also note that the bug that killed round 1 of 1.1.1 was not found
>> >> by any test we currently have. All it would have taken is a test that
>> >> did any map call followed by almost any other bit of javascript (and
>> >> sm 1.7.0).
>> >>
>> >> On 20 October 2011 21:22, Paul Davis <pa...@gmail.com>
>> wrote:
>> >> > On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <
>> randall.leeds@gmail.com>
>> >> wrote:
>> >> >> On Thu, Oct 20, 2011 at 13:42, Paul Davis <
>> paul.joseph.davis@gmail.com
>> >> >wrote:
>> >> >>
>> >> >>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <ra...@apache.org>
>> >> wrote:
>> >> >>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
>> >> >>> lee@projectmastermind.com>wrote:
>> >> >>> >
>> >> >>> >>
>> >> >>> >> > For what it's worth, a CLI based test system is what I was
>> >> imagining
>> >> >>> >> > as well. Take Futon out of the mix and test CouchDB.
>> >> >>> >>
>> >> >>> >> IMO, If CouchDB is intended to be a server that can be accessed
>> from
>> >> >>> >> the browser directly, then there should continue to be some kind
>> of
>> >> >>> >> browser-based test suite that would serve to confirm this
>> >> capability.
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> I have been looking closely at the Futon tests in 1.1.0 for the
>> last
>> >> >>> >> several days, with the idea that I might begin to clean them up a
>> >> bit
>> >> >>> >> as time permits.
>> >> >>> >>
>> >> >>> >> I have found that, while some of these test failures are totally
>> >> bogus,
>> >> >>> >> *some* of them actually do stem from real issues -- minor
>> >> >>> >> incompatibilities between CouchDB's http interface, and the
>> internal
>> >> >>> >> mechanisms of modern browsers (XHR, caching, etc).
>> >> >>> >>
>> >> >>> >> These are problems that we're not going to catch with a
>> stateless,
>> >> >>> >> cache-less http client running on the CLI.  (I can provide
>> examples)
>> >> >>> >>
>> >> >>> >> These issues have the potential to cause real problems for
>> >> >>> >> developers of real browser-based apps "in the wild".  That means,
>> >> >>> >> there's valuable info to be gathered from the browser tests, Iff
>> we
>> >> >>> >> can clean them up, and make them behave consistently; so that
>> >> >>> >> when they fail or succeed, we can actually trust the results.
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> After digging around a good bit, I can see no reason why the
>> >> existing
>> >> >>> >> tests couldn't be cleaned up and made to work correctly in all
>> >> current
>> >> >>> >> versions of major browsers.  I also see no reason why the same
>> tests
>> >> >>> >> couldn't be used successfully from the CLI and `make check` as
>> well.
>> >> >>> >>
>> >> >>> >> I do see significant benefits to using the same javascript test
>> code
>> >> in
>> >> >>> >> all environments we test.
>> >> >>> >>
>> >> >>> >> -Lee
>> >> >>> >> (irc: coltr)
>> >> >>> >>
>> >> >>> >
>> >> >>> >  +1
>> >> >>> > Verify Installation could grow into a suite of browser/futon tests
>> >> that
>> >> >>> > verify that futon (and apps in general) work, including
>> interactions
>> >> with
>> >> >>> > proxies and the like.
>> >> >>>
>> >> >>> Sure. Client tests that test the client are fine.
>> >> >>>
>> >> >>> > The test suite for developers should run cleanly from the CLI as
>> part
>> >> of
>> >> >>> > make check, but continue to be exposed in futon. We should work to
>> be
>> >> >>> sure
>> >> >>> > they function as well as possible, for the reasons you provide.
>> >> >>> >
>> >> >>>
>> >> >>> Blargh no. Server tests should be testing the server. The entire
>> point
>> >> >>> of moving to the command line is so that we don't have to maintain
>> the
>> >> >>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
>> >> >>> release thread) and the wildly varying reports of test output. The
>> >> >>> situation is just a waste of time for everyone involved.
>> >> >>>
>> >> >>> > I think the JS testing situation is a great place for people to
>> jump
>> >> in
>> >> >>> and
>> >> >>> > help out, especially with the browser environment diversity.
>> >> >>> >
>> >> >>>
>> >> >>> Sure, but I don't see what this has to do with browsers.
>> >> >>>
>> >> >>
>> >> >> People who aren't into the internals can help to fix the suite to
>> work
>> >> in
>> >> >> different browser environments. That's all I meant.
>> >> >>
>> >> >
>> >> > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
>> >> > "If these people exist, why do I not see anything in JIRA?"
>> >> >
>> >> >> I suggested that the CLI tests be exposed in Futon because I think
>> there
>> >> are
>> >> >> probably some JS heads in this community who wouldn't have too much
>> >> trouble
>> >> >> fixing a lot of the user agent related issues in the test suite. I
>> >> didn't
>> >> >> mean to suggest that it should continue to be part of the release
>> >> procedure
>> >> >> (it shouldn't) or that we should feel 100% obligated to make sure
>> they
>> >> pass
>> >> >> in 100% of environments (we can't and shouldn't), but J. Lee's point
>> >> about
>> >> >> how keeping such tests around can sometimes expose interesting
>> problems
>> >> we
>> >> >> wouldn't otherwise see, possible outside the CouchDB codebase even,
>> is
>> >> >> worthwhile.
>> >> >>
>> >> >> -Randall
>> >> >>
>> >> >
>> >> > We've had these tests for three years or more now. Perhaps I'm just
>> >> > being dense today but I can't think of a single specific case where
>> >> > testing things in the browser has lead to a bug report/fix that we
>> >> > wouldn't have found with pure CLI tests.
>> >> >
>> >> > The only thing that I'm aware that the tests have done for us is
>> >> > required us to exert a nontrivial amount of effort to keep them
>> >> > running in multiple browser environments. I have no interest in
>> >> > maintaing these as tests runnable in the browser. I want to create a
>> >> > CLI test environment that promotes stable, repeatable, concise tests.
>> >> > Running these in a browser is the antithesis to such an environment.
>> >> >
>> >>
>> >
>>
>

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Noah Slater <ns...@tumbolia.org>]

Wait, did you post that because me and you rock this project like the
Beatles rocked America? Or did you post it with the intention of the song
She Loves You being a sort of meta-commentary on our enviable, and now
infamous, bromance?

On Fri, Oct 21, 2011 at 5:13 AM, Paul Davis <pa...@gmail.com>wrote:

> http://www.youtube.com/watch?feature=player_detailpage&v=G6j5bve7O5E#t=109s
>
> On Thu, Oct 20, 2011 at 7:08 PM, Noah Slater <ns...@tumbolia.org> wrote:
> > +1 on all the stuff Paul said.
> >
> > On Thu, Oct 20, 2011 at 9:25 PM, Robert Newson <rn...@apache.org>
> wrote:
> >
> >> I'll also note that the bug that killed round 1 of 1.1.1 was not found
> >> by any test we currently have. All it would have taken is a test that
> >> did any map call followed by almost any other bit of javascript (and
> >> sm 1.7.0).
> >>
> >> On 20 October 2011 21:22, Paul Davis <pa...@gmail.com>
> wrote:
> >> > On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <
> randall.leeds@gmail.com>
> >> wrote:
> >> >> On Thu, Oct 20, 2011 at 13:42, Paul Davis <
> paul.joseph.davis@gmail.com
> >> >wrote:
> >> >>
> >> >>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <ra...@apache.org>
> >> wrote:
> >> >>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
> >> >>> lee@projectmastermind.com>wrote:
> >> >>> >
> >> >>> >>
> >> >>> >> > For what it's worth, a CLI based test system is what I was
> >> imagining
> >> >>> >> > as well. Take Futon out of the mix and test CouchDB.
> >> >>> >>
> >> >>> >> IMO, If CouchDB is intended to be a server that can be accessed
> from
> >> >>> >> the browser directly, then there should continue to be some kind
> of
> >> >>> >> browser-based test suite that would serve to confirm this
> >> capability.
> >> >>> >>
> >> >>> >>
> >> >>> >> I have been looking closely at the Futon tests in 1.1.0 for the
> last
> >> >>> >> several days, with the idea that I might begin to clean them up a
> >> bit
> >> >>> >> as time permits.
> >> >>> >>
> >> >>> >> I have found that, while some of these test failures are totally
> >> bogus,
> >> >>> >> *some* of them actually do stem from real issues -- minor
> >> >>> >> incompatibilities between CouchDB's http interface, and the
> internal
> >> >>> >> mechanisms of modern browsers (XHR, caching, etc).
> >> >>> >>
> >> >>> >> These are problems that we're not going to catch with a
> stateless,
> >> >>> >> cache-less http client running on the CLI.  (I can provide
> examples)
> >> >>> >>
> >> >>> >> These issues have the potential to cause real problems for
> >> >>> >> developers of real browser-based apps "in the wild".  That means,
> >> >>> >> there's valuable info to be gathered from the browser tests, Iff
> we
> >> >>> >> can clean them up, and make them behave consistently; so that
> >> >>> >> when they fail or succeed, we can actually trust the results.
> >> >>> >>
> >> >>> >>
> >> >>> >> After digging around a good bit, I can see no reason why the
> >> existing
> >> >>> >> tests couldn't be cleaned up and made to work correctly in all
> >> current
> >> >>> >> versions of major browsers.  I also see no reason why the same
> tests
> >> >>> >> couldn't be used successfully from the CLI and `make check` as
> well.
> >> >>> >>
> >> >>> >> I do see significant benefits to using the same javascript test
> code
> >> in
> >> >>> >> all environments we test.
> >> >>> >>
> >> >>> >> -Lee
> >> >>> >> (irc: coltr)
> >> >>> >>
> >> >>> >
> >> >>> >  +1
> >> >>> > Verify Installation could grow into a suite of browser/futon tests
> >> that
> >> >>> > verify that futon (and apps in general) work, including
> interactions
> >> with
> >> >>> > proxies and the like.
> >> >>>
> >> >>> Sure. Client tests that test the client are fine.
> >> >>>
> >> >>> > The test suite for developers should run cleanly from the CLI as
> part
> >> of
> >> >>> > make check, but continue to be exposed in futon. We should work to
> be
> >> >>> sure
> >> >>> > they function as well as possible, for the reasons you provide.
> >> >>> >
> >> >>>
> >> >>> Blargh no. Server tests should be testing the server. The entire
> point
> >> >>> of moving to the command line is so that we don't have to maintain
> the
> >> >>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
> >> >>> release thread) and the wildly varying reports of test output. The
> >> >>> situation is just a waste of time for everyone involved.
> >> >>>
> >> >>> > I think the JS testing situation is a great place for people to
> jump
> >> in
> >> >>> and
> >> >>> > help out, especially with the browser environment diversity.
> >> >>> >
> >> >>>
> >> >>> Sure, but I don't see what this has to do with browsers.
> >> >>>
> >> >>
> >> >> People who aren't into the internals can help to fix the suite to
> work
> >> in
> >> >> different browser environments. That's all I meant.
> >> >>
> >> >
> >> > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
> >> > "If these people exist, why do I not see anything in JIRA?"
> >> >
> >> >> I suggested that the CLI tests be exposed in Futon because I think
> there
> >> are
> >> >> probably some JS heads in this community who wouldn't have too much
> >> trouble
> >> >> fixing a lot of the user agent related issues in the test suite. I
> >> didn't
> >> >> mean to suggest that it should continue to be part of the release
> >> procedure
> >> >> (it shouldn't) or that we should feel 100% obligated to make sure
> they
> >> pass
> >> >> in 100% of environments (we can't and shouldn't), but J. Lee's point
> >> about
> >> >> how keeping such tests around can sometimes expose interesting
> problems
> >> we
> >> >> wouldn't otherwise see, possible outside the CouchDB codebase even,
> is
> >> >> worthwhile.
> >> >>
> >> >> -Randall
> >> >>
> >> >
> >> > We've had these tests for three years or more now. Perhaps I'm just
> >> > being dense today but I can't think of a single specific case where
> >> > testing things in the browser has lead to a bug report/fix that we
> >> > wouldn't have found with pure CLI tests.
> >> >
> >> > The only thing that I'm aware that the tests have done for us is
> >> > required us to exert a nontrivial amount of effort to keep them
> >> > running in multiple browser environments. I have no interest in
> >> > maintaing these as tests runnable in the browser. I want to create a
> >> > CLI test environment that promotes stable, repeatable, concise tests.
> >> > Running these in a browser is the antithesis to such an environment.
> >> >
> >>
> >
>

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Paul Davis <pa...@gmail.com>]

http://www.youtube.com/watch?feature=player_detailpage&v=G6j5bve7O5E#t=109s

On Thu, Oct 20, 2011 at 7:08 PM, Noah Slater <ns...@tumbolia.org> wrote:
> +1 on all the stuff Paul said.
>
> On Thu, Oct 20, 2011 at 9:25 PM, Robert Newson <rn...@apache.org> wrote:
>
>> I'll also note that the bug that killed round 1 of 1.1.1 was not found
>> by any test we currently have. All it would have taken is a test that
>> did any map call followed by almost any other bit of javascript (and
>> sm 1.7.0).
>>
>> On 20 October 2011 21:22, Paul Davis <pa...@gmail.com> wrote:
>> > On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <ra...@gmail.com>
>> wrote:
>> >> On Thu, Oct 20, 2011 at 13:42, Paul Davis <paul.joseph.davis@gmail.com
>> >wrote:
>> >>
>> >>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <ra...@apache.org>
>> wrote:
>> >>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
>> >>> lee@projectmastermind.com>wrote:
>> >>> >
>> >>> >>
>> >>> >> > For what it's worth, a CLI based test system is what I was
>> imagining
>> >>> >> > as well. Take Futon out of the mix and test CouchDB.
>> >>> >>
>> >>> >> IMO, If CouchDB is intended to be a server that can be accessed from
>> >>> >> the browser directly, then there should continue to be some kind of
>> >>> >> browser-based test suite that would serve to confirm this
>> capability.
>> >>> >>
>> >>> >>
>> >>> >> I have been looking closely at the Futon tests in 1.1.0 for the last
>> >>> >> several days, with the idea that I might begin to clean them up a
>> bit
>> >>> >> as time permits.
>> >>> >>
>> >>> >> I have found that, while some of these test failures are totally
>> bogus,
>> >>> >> *some* of them actually do stem from real issues -- minor
>> >>> >> incompatibilities between CouchDB's http interface, and the internal
>> >>> >> mechanisms of modern browsers (XHR, caching, etc).
>> >>> >>
>> >>> >> These are problems that we're not going to catch with a stateless,
>> >>> >> cache-less http client running on the CLI.  (I can provide examples)
>> >>> >>
>> >>> >> These issues have the potential to cause real problems for
>> >>> >> developers of real browser-based apps "in the wild".  That means,
>> >>> >> there's valuable info to be gathered from the browser tests, Iff we
>> >>> >> can clean them up, and make them behave consistently; so that
>> >>> >> when they fail or succeed, we can actually trust the results.
>> >>> >>
>> >>> >>
>> >>> >> After digging around a good bit, I can see no reason why the
>> existing
>> >>> >> tests couldn't be cleaned up and made to work correctly in all
>> current
>> >>> >> versions of major browsers.  I also see no reason why the same tests
>> >>> >> couldn't be used successfully from the CLI and `make check` as well.
>> >>> >>
>> >>> >> I do see significant benefits to using the same javascript test code
>> in
>> >>> >> all environments we test.
>> >>> >>
>> >>> >> -Lee
>> >>> >> (irc: coltr)
>> >>> >>
>> >>> >
>> >>> >  +1
>> >>> > Verify Installation could grow into a suite of browser/futon tests
>> that
>> >>> > verify that futon (and apps in general) work, including interactions
>> with
>> >>> > proxies and the like.
>> >>>
>> >>> Sure. Client tests that test the client are fine.
>> >>>
>> >>> > The test suite for developers should run cleanly from the CLI as part
>> of
>> >>> > make check, but continue to be exposed in futon. We should work to be
>> >>> sure
>> >>> > they function as well as possible, for the reasons you provide.
>> >>> >
>> >>>
>> >>> Blargh no. Server tests should be testing the server. The entire point
>> >>> of moving to the command line is so that we don't have to maintain the
>> >>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
>> >>> release thread) and the wildly varying reports of test output. The
>> >>> situation is just a waste of time for everyone involved.
>> >>>
>> >>> > I think the JS testing situation is a great place for people to jump
>> in
>> >>> and
>> >>> > help out, especially with the browser environment diversity.
>> >>> >
>> >>>
>> >>> Sure, but I don't see what this has to do with browsers.
>> >>>
>> >>
>> >> People who aren't into the internals can help to fix the suite to work
>> in
>> >> different browser environments. That's all I meant.
>> >>
>> >
>> > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
>> > "If these people exist, why do I not see anything in JIRA?"
>> >
>> >> I suggested that the CLI tests be exposed in Futon because I think there
>> are
>> >> probably some JS heads in this community who wouldn't have too much
>> trouble
>> >> fixing a lot of the user agent related issues in the test suite. I
>> didn't
>> >> mean to suggest that it should continue to be part of the release
>> procedure
>> >> (it shouldn't) or that we should feel 100% obligated to make sure they
>> pass
>> >> in 100% of environments (we can't and shouldn't), but J. Lee's point
>> about
>> >> how keeping such tests around can sometimes expose interesting problems
>> we
>> >> wouldn't otherwise see, possible outside the CouchDB codebase even, is
>> >> worthwhile.
>> >>
>> >> -Randall
>> >>
>> >
>> > We've had these tests for three years or more now. Perhaps I'm just
>> > being dense today but I can't think of a single specific case where
>> > testing things in the browser has lead to a bug report/fix that we
>> > wouldn't have found with pure CLI tests.
>> >
>> > The only thing that I'm aware that the tests have done for us is
>> > required us to exert a nontrivial amount of effort to keep them
>> > running in multiple browser environments. I have no interest in
>> > maintaing these as tests runnable in the browser. I want to create a
>> > CLI test environment that promotes stable, repeatable, concise tests.
>> > Running these in a browser is the antithesis to such an environment.
>> >
>>
>

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Noah Slater <ns...@tumbolia.org>]

+1 on all the stuff Paul said.

On Thu, Oct 20, 2011 at 9:25 PM, Robert Newson <rn...@apache.org> wrote:

> I'll also note that the bug that killed round 1 of 1.1.1 was not found
> by any test we currently have. All it would have taken is a test that
> did any map call followed by almost any other bit of javascript (and
> sm 1.7.0).
>
> On 20 October 2011 21:22, Paul Davis <pa...@gmail.com> wrote:
> > On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <ra...@gmail.com>
> wrote:
> >> On Thu, Oct 20, 2011 at 13:42, Paul Davis <paul.joseph.davis@gmail.com
> >wrote:
> >>
> >>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <ra...@apache.org>
> wrote:
> >>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
> >>> lee@projectmastermind.com>wrote:
> >>> >
> >>> >>
> >>> >> > For what it's worth, a CLI based test system is what I was
> imagining
> >>> >> > as well. Take Futon out of the mix and test CouchDB.
> >>> >>
> >>> >> IMO, If CouchDB is intended to be a server that can be accessed from
> >>> >> the browser directly, then there should continue to be some kind of
> >>> >> browser-based test suite that would serve to confirm this
> capability.
> >>> >>
> >>> >>
> >>> >> I have been looking closely at the Futon tests in 1.1.0 for the last
> >>> >> several days, with the idea that I might begin to clean them up a
> bit
> >>> >> as time permits.
> >>> >>
> >>> >> I have found that, while some of these test failures are totally
> bogus,
> >>> >> *some* of them actually do stem from real issues -- minor
> >>> >> incompatibilities between CouchDB's http interface, and the internal
> >>> >> mechanisms of modern browsers (XHR, caching, etc).
> >>> >>
> >>> >> These are problems that we're not going to catch with a stateless,
> >>> >> cache-less http client running on the CLI.  (I can provide examples)
> >>> >>
> >>> >> These issues have the potential to cause real problems for
> >>> >> developers of real browser-based apps "in the wild".  That means,
> >>> >> there's valuable info to be gathered from the browser tests, Iff we
> >>> >> can clean them up, and make them behave consistently; so that
> >>> >> when they fail or succeed, we can actually trust the results.
> >>> >>
> >>> >>
> >>> >> After digging around a good bit, I can see no reason why the
> existing
> >>> >> tests couldn't be cleaned up and made to work correctly in all
> current
> >>> >> versions of major browsers.  I also see no reason why the same tests
> >>> >> couldn't be used successfully from the CLI and `make check` as well.
> >>> >>
> >>> >> I do see significant benefits to using the same javascript test code
> in
> >>> >> all environments we test.
> >>> >>
> >>> >> -Lee
> >>> >> (irc: coltr)
> >>> >>
> >>> >
> >>> >  +1
> >>> > Verify Installation could grow into a suite of browser/futon tests
> that
> >>> > verify that futon (and apps in general) work, including interactions
> with
> >>> > proxies and the like.
> >>>
> >>> Sure. Client tests that test the client are fine.
> >>>
> >>> > The test suite for developers should run cleanly from the CLI as part
> of
> >>> > make check, but continue to be exposed in futon. We should work to be
> >>> sure
> >>> > they function as well as possible, for the reasons you provide.
> >>> >
> >>>
> >>> Blargh no. Server tests should be testing the server. The entire point
> >>> of moving to the command line is so that we don't have to maintain the
> >>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
> >>> release thread) and the wildly varying reports of test output. The
> >>> situation is just a waste of time for everyone involved.
> >>>
> >>> > I think the JS testing situation is a great place for people to jump
> in
> >>> and
> >>> > help out, especially with the browser environment diversity.
> >>> >
> >>>
> >>> Sure, but I don't see what this has to do with browsers.
> >>>
> >>
> >> People who aren't into the internals can help to fix the suite to work
> in
> >> different browser environments. That's all I meant.
> >>
> >
> > Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
> > "If these people exist, why do I not see anything in JIRA?"
> >
> >> I suggested that the CLI tests be exposed in Futon because I think there
> are
> >> probably some JS heads in this community who wouldn't have too much
> trouble
> >> fixing a lot of the user agent related issues in the test suite. I
> didn't
> >> mean to suggest that it should continue to be part of the release
> procedure
> >> (it shouldn't) or that we should feel 100% obligated to make sure they
> pass
> >> in 100% of environments (we can't and shouldn't), but J. Lee's point
> about
> >> how keeping such tests around can sometimes expose interesting problems
> we
> >> wouldn't otherwise see, possible outside the CouchDB codebase even, is
> >> worthwhile.
> >>
> >> -Randall
> >>
> >
> > We've had these tests for three years or more now. Perhaps I'm just
> > being dense today but I can't think of a single specific case where
> > testing things in the browser has lead to a bug report/fix that we
> > wouldn't have found with pure CLI tests.
> >
> > The only thing that I'm aware that the tests have done for us is
> > required us to exert a nontrivial amount of effort to keep them
> > running in multiple browser environments. I have no interest in
> > maintaing these as tests runnable in the browser. I want to create a
> > CLI test environment that promotes stable, repeatable, concise tests.
> > Running these in a browser is the antithesis to such an environment.
> >
>

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Robert Newson <rn...@apache.org>]

I'll also note that the bug that killed round 1 of 1.1.1 was not found
by any test we currently have. All it would have taken is a test that
did any map call followed by almost any other bit of javascript (and
sm 1.7.0).

On 20 October 2011 21:22, Paul Davis <pa...@gmail.com> wrote:
> On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <ra...@gmail.com> wrote:
>> On Thu, Oct 20, 2011 at 13:42, Paul Davis <pa...@gmail.com>wrote:
>>
>>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <ra...@apache.org> wrote:
>>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
>>> lee@projectmastermind.com>wrote:
>>> >
>>> >>
>>> >> > For what it's worth, a CLI based test system is what I was imagining
>>> >> > as well. Take Futon out of the mix and test CouchDB.
>>> >>
>>> >> IMO, If CouchDB is intended to be a server that can be accessed from
>>> >> the browser directly, then there should continue to be some kind of
>>> >> browser-based test suite that would serve to confirm this capability.
>>> >>
>>> >>
>>> >> I have been looking closely at the Futon tests in 1.1.0 for the last
>>> >> several days, with the idea that I might begin to clean them up a bit
>>> >> as time permits.
>>> >>
>>> >> I have found that, while some of these test failures are totally bogus,
>>> >> *some* of them actually do stem from real issues -- minor
>>> >> incompatibilities between CouchDB's http interface, and the internal
>>> >> mechanisms of modern browsers (XHR, caching, etc).
>>> >>
>>> >> These are problems that we're not going to catch with a stateless,
>>> >> cache-less http client running on the CLI.  (I can provide examples)
>>> >>
>>> >> These issues have the potential to cause real problems for
>>> >> developers of real browser-based apps "in the wild".  That means,
>>> >> there's valuable info to be gathered from the browser tests, Iff we
>>> >> can clean them up, and make them behave consistently; so that
>>> >> when they fail or succeed, we can actually trust the results.
>>> >>
>>> >>
>>> >> After digging around a good bit, I can see no reason why the existing
>>> >> tests couldn't be cleaned up and made to work correctly in all current
>>> >> versions of major browsers.  I also see no reason why the same tests
>>> >> couldn't be used successfully from the CLI and `make check` as well.
>>> >>
>>> >> I do see significant benefits to using the same javascript test code in
>>> >> all environments we test.
>>> >>
>>> >> -Lee
>>> >> (irc: coltr)
>>> >>
>>> >
>>> >  +1
>>> > Verify Installation could grow into a suite of browser/futon tests that
>>> > verify that futon (and apps in general) work, including interactions with
>>> > proxies and the like.
>>>
>>> Sure. Client tests that test the client are fine.
>>>
>>> > The test suite for developers should run cleanly from the CLI as part of
>>> > make check, but continue to be exposed in futon. We should work to be
>>> sure
>>> > they function as well as possible, for the reasons you provide.
>>> >
>>>
>>> Blargh no. Server tests should be testing the server. The entire point
>>> of moving to the command line is so that we don't have to maintain the
>>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
>>> release thread) and the wildly varying reports of test output. The
>>> situation is just a waste of time for everyone involved.
>>>
>>> > I think the JS testing situation is a great place for people to jump in
>>> and
>>> > help out, especially with the browser environment diversity.
>>> >
>>>
>>> Sure, but I don't see what this has to do with browsers.
>>>
>>
>> People who aren't into the internals can help to fix the suite to work in
>> different browser environments. That's all I meant.
>>
>
> Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
> "If these people exist, why do I not see anything in JIRA?"
>
>> I suggested that the CLI tests be exposed in Futon because I think there are
>> probably some JS heads in this community who wouldn't have too much trouble
>> fixing a lot of the user agent related issues in the test suite. I didn't
>> mean to suggest that it should continue to be part of the release procedure
>> (it shouldn't) or that we should feel 100% obligated to make sure they pass
>> in 100% of environments (we can't and shouldn't), but J. Lee's point about
>> how keeping such tests around can sometimes expose interesting problems we
>> wouldn't otherwise see, possible outside the CouchDB codebase even, is
>> worthwhile.
>>
>> -Randall
>>
>
> We've had these tests for three years or more now. Perhaps I'm just
> being dense today but I can't think of a single specific case where
> testing things in the browser has lead to a bug report/fix that we
> wouldn't have found with pure CLI tests.
>
> The only thing that I'm aware that the tests have done for us is
> required us to exert a nontrivial amount of effort to keep them
> running in multiple browser environments. I have no interest in
> maintaing these as tests runnable in the browser. I want to create a
> CLI test environment that promotes stable, repeatable, concise tests.
> Running these in a browser is the antithesis to such an environment.
>

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Paul Davis <pa...@gmail.com>]

On Thu, Oct 20, 2011 at 1:45 PM, Randall Leeds <ra...@gmail.com> wrote:
> On Thu, Oct 20, 2011 at 13:42, Paul Davis <pa...@gmail.com>wrote:
>
>> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <ra...@apache.org> wrote:
>> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
>> lee@projectmastermind.com>wrote:
>> >
>> >>
>> >> > For what it's worth, a CLI based test system is what I was imagining
>> >> > as well. Take Futon out of the mix and test CouchDB.
>> >>
>> >> IMO, If CouchDB is intended to be a server that can be accessed from
>> >> the browser directly, then there should continue to be some kind of
>> >> browser-based test suite that would serve to confirm this capability.
>> >>
>> >>
>> >> I have been looking closely at the Futon tests in 1.1.0 for the last
>> >> several days, with the idea that I might begin to clean them up a bit
>> >> as time permits.
>> >>
>> >> I have found that, while some of these test failures are totally bogus,
>> >> *some* of them actually do stem from real issues -- minor
>> >> incompatibilities between CouchDB's http interface, and the internal
>> >> mechanisms of modern browsers (XHR, caching, etc).
>> >>
>> >> These are problems that we're not going to catch with a stateless,
>> >> cache-less http client running on the CLI.  (I can provide examples)
>> >>
>> >> These issues have the potential to cause real problems for
>> >> developers of real browser-based apps "in the wild".  That means,
>> >> there's valuable info to be gathered from the browser tests, Iff we
>> >> can clean them up, and make them behave consistently; so that
>> >> when they fail or succeed, we can actually trust the results.
>> >>
>> >>
>> >> After digging around a good bit, I can see no reason why the existing
>> >> tests couldn't be cleaned up and made to work correctly in all current
>> >> versions of major browsers.  I also see no reason why the same tests
>> >> couldn't be used successfully from the CLI and `make check` as well.
>> >>
>> >> I do see significant benefits to using the same javascript test code in
>> >> all environments we test.
>> >>
>> >> -Lee
>> >> (irc: coltr)
>> >>
>> >
>> >  +1
>> > Verify Installation could grow into a suite of browser/futon tests that
>> > verify that futon (and apps in general) work, including interactions with
>> > proxies and the like.
>>
>> Sure. Client tests that test the client are fine.
>>
>> > The test suite for developers should run cleanly from the CLI as part of
>> > make check, but continue to be exposed in futon. We should work to be
>> sure
>> > they function as well as possible, for the reasons you provide.
>> >
>>
>> Blargh no. Server tests should be testing the server. The entire point
>> of moving to the command line is so that we don't have to maintain the
>> Futon test suite. Just look at the 1.1.1 thread (or damn near any
>> release thread) and the wildly varying reports of test output. The
>> situation is just a waste of time for everyone involved.
>>
>> > I think the JS testing situation is a great place for people to jump in
>> and
>> > help out, especially with the browser environment diversity.
>> >
>>
>> Sure, but I don't see what this has to do with browsers.
>>
>
> People who aren't into the internals can help to fix the suite to work in
> different browser environments. That's all I meant.
>

Seeing as I'm having a Negative Nancy day, I'll just ask rhetorically,
"If these people exist, why do I not see anything in JIRA?"

> I suggested that the CLI tests be exposed in Futon because I think there are
> probably some JS heads in this community who wouldn't have too much trouble
> fixing a lot of the user agent related issues in the test suite. I didn't
> mean to suggest that it should continue to be part of the release procedure
> (it shouldn't) or that we should feel 100% obligated to make sure they pass
> in 100% of environments (we can't and shouldn't), but J. Lee's point about
> how keeping such tests around can sometimes expose interesting problems we
> wouldn't otherwise see, possible outside the CouchDB codebase even, is
> worthwhile.
>
> -Randall
>

We've had these tests for three years or more now. Perhaps I'm just
being dense today but I can't think of a single specific case where
testing things in the browser has lead to a bug report/fix that we
wouldn't have found with pure CLI tests.

The only thing that I'm aware that the tests have done for us is
required us to exert a nontrivial amount of effort to keep them
running in multiple browser environments. I have no interest in
maintaing these as tests runnable in the browser. I want to create a
CLI test environment that promotes stable, repeatable, concise tests.
Running these in a browser is the antithesis to such an environment.

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Randall Leeds <ra...@gmail.com>]

On Thu, Oct 20, 2011 at 13:42, Paul Davis <pa...@gmail.com>wrote:

> On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <ra...@apache.org> wrote:
> > On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <
> lee@projectmastermind.com>wrote:
> >
> >>
> >> > For what it's worth, a CLI based test system is what I was imagining
> >> > as well. Take Futon out of the mix and test CouchDB.
> >>
> >> IMO, If CouchDB is intended to be a server that can be accessed from
> >> the browser directly, then there should continue to be some kind of
> >> browser-based test suite that would serve to confirm this capability.
> >>
> >>
> >> I have been looking closely at the Futon tests in 1.1.0 for the last
> >> several days, with the idea that I might begin to clean them up a bit
> >> as time permits.
> >>
> >> I have found that, while some of these test failures are totally bogus,
> >> *some* of them actually do stem from real issues -- minor
> >> incompatibilities between CouchDB's http interface, and the internal
> >> mechanisms of modern browsers (XHR, caching, etc).
> >>
> >> These are problems that we're not going to catch with a stateless,
> >> cache-less http client running on the CLI.  (I can provide examples)
> >>
> >> These issues have the potential to cause real problems for
> >> developers of real browser-based apps "in the wild".  That means,
> >> there's valuable info to be gathered from the browser tests, Iff we
> >> can clean them up, and make them behave consistently; so that
> >> when they fail or succeed, we can actually trust the results.
> >>
> >>
> >> After digging around a good bit, I can see no reason why the existing
> >> tests couldn't be cleaned up and made to work correctly in all current
> >> versions of major browsers.  I also see no reason why the same tests
> >> couldn't be used successfully from the CLI and `make check` as well.
> >>
> >> I do see significant benefits to using the same javascript test code in
> >> all environments we test.
> >>
> >> -Lee
> >> (irc: coltr)
> >>
> >
> >  +1
> > Verify Installation could grow into a suite of browser/futon tests that
> > verify that futon (and apps in general) work, including interactions with
> > proxies and the like.
>
> Sure. Client tests that test the client are fine.
>
> > The test suite for developers should run cleanly from the CLI as part of
> > make check, but continue to be exposed in futon. We should work to be
> sure
> > they function as well as possible, for the reasons you provide.
> >
>
> Blargh no. Server tests should be testing the server. The entire point
> of moving to the command line is so that we don't have to maintain the
> Futon test suite. Just look at the 1.1.1 thread (or damn near any
> release thread) and the wildly varying reports of test output. The
> situation is just a waste of time for everyone involved.
>
> > I think the JS testing situation is a great place for people to jump in
> and
> > help out, especially with the browser environment diversity.
> >
>
> Sure, but I don't see what this has to do with browsers.
>

People who aren't into the internals can help to fix the suite to work in
different browser environments. That's all I meant.

I suggested that the CLI tests be exposed in Futon because I think there are
probably some JS heads in this community who wouldn't have too much trouble
fixing a lot of the user agent related issues in the test suite. I didn't
mean to suggest that it should continue to be part of the release procedure
(it shouldn't) or that we should feel 100% obligated to make sure they pass
in 100% of environments (we can't and shouldn't), but J. Lee's point about
how keeping such tests around can sometimes expose interesting problems we
wouldn't otherwise see, possible outside the CouchDB codebase even, is
worthwhile.

-Randall

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Paul Davis <pa...@gmail.com>]

On Thu, Oct 20, 2011 at 1:38 AM, Randall Leeds <ra...@apache.org> wrote:
> On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <le...@projectmastermind.com>wrote:
>
>>
>> > For what it's worth, a CLI based test system is what I was imagining
>> > as well. Take Futon out of the mix and test CouchDB.
>>
>> IMO, If CouchDB is intended to be a server that can be accessed from
>> the browser directly, then there should continue to be some kind of
>> browser-based test suite that would serve to confirm this capability.
>>
>>
>> I have been looking closely at the Futon tests in 1.1.0 for the last
>> several days, with the idea that I might begin to clean them up a bit
>> as time permits.
>>
>> I have found that, while some of these test failures are totally bogus,
>> *some* of them actually do stem from real issues -- minor
>> incompatibilities between CouchDB's http interface, and the internal
>> mechanisms of modern browsers (XHR, caching, etc).
>>
>> These are problems that we're not going to catch with a stateless,
>> cache-less http client running on the CLI.  (I can provide examples)
>>
>> These issues have the potential to cause real problems for
>> developers of real browser-based apps "in the wild".  That means,
>> there's valuable info to be gathered from the browser tests, Iff we
>> can clean them up, and make them behave consistently; so that
>> when they fail or succeed, we can actually trust the results.
>>
>>
>> After digging around a good bit, I can see no reason why the existing
>> tests couldn't be cleaned up and made to work correctly in all current
>> versions of major browsers.  I also see no reason why the same tests
>> couldn't be used successfully from the CLI and `make check` as well.
>>
>> I do see significant benefits to using the same javascript test code in
>> all environments we test.
>>
>> -Lee
>> (irc: coltr)
>>
>
>  +1
> Verify Installation could grow into a suite of browser/futon tests that
> verify that futon (and apps in general) work, including interactions with
> proxies and the like.

Sure. Client tests that test the client are fine.

> The test suite for developers should run cleanly from the CLI as part of
> make check, but continue to be exposed in futon. We should work to be sure
> they function as well as possible, for the reasons you provide.
>

Blargh no. Server tests should be testing the server. The entire point
of moving to the command line is so that we don't have to maintain the
Futon test suite. Just look at the 1.1.1 thread (or damn near any
release thread) and the wildly varying reports of test output. The
situation is just a waste of time for everyone involved.

> I think the JS testing situation is a great place for people to jump in and
> help out, especially with the browser environment diversity.
>

Sure, but I don't see what this has to do with browsers.

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Randall Leeds <ra...@apache.org>]

On Wed, Oct 19, 2011 at 23:38, J. Lee Coltrane <le...@projectmastermind.com>wrote:

>
> > For what it's worth, a CLI based test system is what I was imagining
> > as well. Take Futon out of the mix and test CouchDB.
>
> IMO, If CouchDB is intended to be a server that can be accessed from
> the browser directly, then there should continue to be some kind of
> browser-based test suite that would serve to confirm this capability.
>
>
> I have been looking closely at the Futon tests in 1.1.0 for the last
> several days, with the idea that I might begin to clean them up a bit
> as time permits.
>
> I have found that, while some of these test failures are totally bogus,
> *some* of them actually do stem from real issues -- minor
> incompatibilities between CouchDB's http interface, and the internal
> mechanisms of modern browsers (XHR, caching, etc).
>
> These are problems that we're not going to catch with a stateless,
> cache-less http client running on the CLI.  (I can provide examples)
>
> These issues have the potential to cause real problems for
> developers of real browser-based apps "in the wild".  That means,
> there's valuable info to be gathered from the browser tests, Iff we
> can clean them up, and make them behave consistently; so that
> when they fail or succeed, we can actually trust the results.
>
>
> After digging around a good bit, I can see no reason why the existing
> tests couldn't be cleaned up and made to work correctly in all current
> versions of major browsers.  I also see no reason why the same tests
> couldn't be used successfully from the CLI and `make check` as well.
>
> I do see significant benefits to using the same javascript test code in
> all environments we test.
>
> -Lee
> (irc: coltr)
>

 +1
Verify Installation could grow into a suite of browser/futon tests that
verify that futon (and apps in general) work, including interactions with
proxies and the like.
The test suite for developers should run cleanly from the CLI as part of
make check, but continue to be exposed in futon. We should work to be sure
they function as well as possible, for the reasons you provide.

I think the JS testing situation is a great place for people to jump in and
help out, especially with the browser environment diversity.

-R

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Klaus Trainer <kl...@posteo.de>]

+1


On Wed, 2011-10-19 at 23:38 -0400, J. Lee Coltrane wrote:
> > For what it's worth, a CLI based test system is what I was imagining
> > as well. Take Futon out of the mix and test CouchDB.
> 
> IMO, If CouchDB is intended to be a server that can be accessed from
> the browser directly, then there should continue to be some kind of 
> browser-based test suite that would serve to confirm this capability.  
> 
> 
> I have been looking closely at the Futon tests in 1.1.0 for the last
> several days, with the idea that I might begin to clean them up a bit
> as time permits.
> 
> I have found that, while some of these test failures are totally bogus, 
> *some* of them actually do stem from real issues -- minor 
> incompatibilities between CouchDB's http interface, and the internal 
> mechanisms of modern browsers (XHR, caching, etc).
> 
> These are problems that we're not going to catch with a stateless, 
> cache-less http client running on the CLI.  (I can provide examples)
> 
> These issues have the potential to cause real problems for 
> developers of real browser-based apps "in the wild".  That means, 
> there's valuable info to be gathered from the browser tests, Iff we 
> can clean them up, and make them behave consistently; so that 
> when they fail or succeed, we can actually trust the results.
> 
> 
> After digging around a good bit, I can see no reason why the existing 
> tests couldn't be cleaned up and made to work correctly in all current 
> versions of major browsers.  I also see no reason why the same tests 
> couldn't be used successfully from the CLI and `make check` as well.
> 
> I do see significant benefits to using the same javascript test code in
> all environments we test.
> 
> -Lee
> (irc: coltr)

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

["J. Lee Coltrane" <le...@projectmastermind.com>]

> For what it's worth, a CLI based test system is what I was imagining
> as well. Take Futon out of the mix and test CouchDB.

IMO, If CouchDB is intended to be a server that can be accessed from
the browser directly, then there should continue to be some kind of 
browser-based test suite that would serve to confirm this capability.  


I have been looking closely at the Futon tests in 1.1.0 for the last
several days, with the idea that I might begin to clean them up a bit
as time permits.

I have found that, while some of these test failures are totally bogus, 
*some* of them actually do stem from real issues -- minor 
incompatibilities between CouchDB's http interface, and the internal 
mechanisms of modern browsers (XHR, caching, etc).

These are problems that we're not going to catch with a stateless, 
cache-less http client running on the CLI.  (I can provide examples)

These issues have the potential to cause real problems for 
developers of real browser-based apps "in the wild".  That means, 
there's valuable info to be gathered from the browser tests, Iff we 
can clean them up, and make them behave consistently; so that 
when they fail or succeed, we can actually trust the results.


After digging around a good bit, I can see no reason why the existing 
tests couldn't be cleaned up and made to work correctly in all current 
versions of major browsers.  I also see no reason why the same tests 
couldn't be used successfully from the CLI and `make check` as well.

I do see significant benefits to using the same javascript test code in
all environments we test.

-Lee
(irc: coltr)

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Sam Bisbee <sa...@sbisbee.com>]

On Wed, Oct 19, 2011 at 6:12 PM, Paul Davis <pa...@gmail.com> wrote:
> On Wed, Oct 19, 2011 at 5:00 PM, Sam Bisbee <sa...@sbisbee.com> wrote:
>> On Wed, Oct 19, 2011 at 3:52 PM, Jan Lehnardt <ja...@apache.org> wrote:
>>>
>>> On Oct 19, 2011, at 21:25 , Sam Bisbee wrote:
>>>
>>>> Yeah, it looks like it's just an unstable test. Which makes me wonder
>>>> why we even have it or include it in the release process.
>>>
>>> It is only unstable in certain environments and we are constantly improving all tests to be more stable in all environments, but the browser world is harsh :)
>>
>> I'm all for improvement. :) But is this improvement being tracked
>> anywhere, JIRA or otherwise? And if not, then could it be? I'm sure
>> that folks (self included) would be willing to pitch in if we knew
>> what had to be done.
>>
>
> There was a thread about splitting Futon tests out and making them
> proper CLI tests that are run as part of `make check` and become
> actual release blockers. I have often and loudly voiced my support of
> this approach. I refer you to the number of varying reports for 1.1.1
> with various browsers as Exhibit A in favor of this switch.

Yeah, that sounds like the thread I was remembering.

For what it's worth, a CLI based test system is what I was imagining
as well. Take Futon out of the mix and test CouchDB.

>>> I'd still consider having the test better than not having it at all :)
>>
>> I semi remember this being discussed before, but are the Futon tests
>> meant to test CouchDB, Futon, or the integration of the two?
>>
>
> The test suite is there to test CouchDB. There's some merit to testing
> Futon/Browser access as well. Jan's recent approach to the "verify
> installation" seems to me like a good way forward. We could use a
> small number of tests that check various browser capabilities and what
> not. But I think a large amount of the Futon test code should be
> permanently removed from the browser based test runner.

I think that there is merit in unit testing futon.js and
jquery.couch.js, which just makes those things sound more like their
own project. As a side, semi related note I have always wondered why
jquery.couch.js was part of Apache CouchDB.

For browser/Futon checks, it seems to me that these are more of
environmental checks: is the right version of jquery being used, is it
a supported browser, etc.

Verifying the installation is running `make check`, turning on the
server (which should be checking more permissions), and then running
your software.

Cheers,

--
Sam Bisbee

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Randall Leeds <ra...@gmail.com>]

On Wed, Oct 19, 2011 at 18:12, Paul Davis <pa...@gmail.com>wrote:

> On Wed, Oct 19, 2011 at 5:00 PM, Sam Bisbee <sa...@sbisbee.com> wrote:
> > On Wed, Oct 19, 2011 at 3:52 PM, Jan Lehnardt <ja...@apache.org> wrote:
> >>
> >> On Oct 19, 2011, at 21:25 , Sam Bisbee wrote:
> >>
> >>> Yeah, it looks like it's just an unstable test. Which makes me wonder
> >>> why we even have it or include it in the release process.
> >>
> >> It is only unstable in certain environments and we are constantly
> improving all tests to be more stable in all environments, but the browser
> world is harsh :)
> >
> > I'm all for improvement. :) But is this improvement being tracked
> > anywhere, JIRA or otherwise? And if not, then could it be? I'm sure
> > that folks (self included) would be willing to pitch in if we knew
> > what had to be done.
> >
>

> There was a thread about splitting Futon tests out and making them
> proper CLI tests that are run as part of `make check` and become
> actual release blockers. I have often and loudly voiced my support of
> this approach. I refer you to the number of varying reports for 1.1.1
> with various browsers as Exhibit A in favor of this switch.
>

+1
I opened an issue and assigned it to me:
https://issues.apache.org/jira/browse/COUCHDB-1312

-Randall

Re: futon tests (Was: Re: [VOTE] Apache CouchDB 1.1.1 Release)

[Paul Davis <pa...@gmail.com>]

On Wed, Oct 19, 2011 at 5:00 PM, Sam Bisbee <sa...@sbisbee.com> wrote:
> On Wed, Oct 19, 2011 at 3:52 PM, Jan Lehnardt <ja...@apache.org> wrote:
>>
>> On Oct 19, 2011, at 21:25 , Sam Bisbee wrote:
>>
>>> Yeah, it looks like it's just an unstable test. Which makes me wonder
>>> why we even have it or include it in the release process.
>>
>> It is only unstable in certain environments and we are constantly improving all tests to be more stable in all environments, but the browser world is harsh :)
>
> I'm all for improvement. :) But is this improvement being tracked
> anywhere, JIRA or otherwise? And if not, then could it be? I'm sure
> that folks (self included) would be willing to pitch in if we knew
> what had to be done.
>

There was a thread about splitting Futon tests out and making them
proper CLI tests that are run as part of `make check` and become
actual release blockers. I have often and loudly voiced my support of
this approach. I refer you to the number of varying reports for 1.1.1
with various browsers as Exhibit A in favor of this switch.

>> I'd still consider having the test better than not having it at all :)
>
> I semi remember this being discussed before, but are the Futon tests
> meant to test CouchDB, Futon, or the integration of the two?
>

The test suite is there to test CouchDB. There's some merit to testing
Futon/Browser access as well. Jan's recent approach to the "verify
installation" seems to me like a good way forward. We could use a
small number of tests that check various browser capabilities and what
not. But I think a large amount of the Futon test code should be
permanently removed from the browser based test runner.

> Cheers,
>
> --
> Sam Bisbee
>