You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cordova.apache.org by Shazron <sh...@gmail.com> on 2014/10/30 21:54:16 UTC
[mobile-spec] consistency and reliability
`cordova-mobile-spec/createmobilespec/createmobilespec.js --ios
--globalplugins`
Has anyone gotten mobile-spec to work consistently? I get wildly different
results on each run. Sometimes there is more than one report in a run. See:
[image: Inline image 1]
Re: [mobile-spec] consistency and reliability
Posted by Shazron <sh...@gmail.com>.
Brian - I looked into tap and someone has wrote a Jasmine tap reporter (so
we can just plug this in) -- but the usage problems that Michal wrote about
won't just fix themselves I don't think if we use the tap reporter so there
is still work to be done just getting it to work reliably.
On Thu, Oct 30, 2014 at 5:09 PM, Brian LeRoux <b...@brian.io> wrote:
> Nope.
>
> On Thu, Oct 30, 2014, 5:04 PM Jesse <pu...@gmail.com> wrote:
>
> > I would much rather we fix things, than continually rewrite + discard,
> > which seems to be the norm these days.
> > tape/tap would require us to throw away thousands of jasmine2 based tests
> > wouldn't it?
> >
> >
> > @purplecabbage
> > risingj.com
> >
> > On Thu, Oct 30, 2014 at 4:44 PM, Brian LeRoux <b...@brian.io> wrote:
> >
> > > tape (and or tap) has no globals and simple / shallow callback solution
> > > which fixes that
> > >
> > > we will have fresh intern blood to sacrifice to the test lib gods alter
> > > soon..
> > >
> > > On Thu, Oct 30, 2014, 4:10 PM Michal Mocny <mm...@chromium.org>
> wrote:
> > >
> > > > I've seen that in the past and it usually happens due to misuse of
> > > jasmine,
> > > > which seems unfortunately quite brittle. Specifically, if done()
> isn't
> > > > called, or an async event fires after a test has already failed /
> timed
> > > out
> > > > and calls done(), or registers new tests with it() etc, weird things
> > will
> > > > happen.
> > > >
> > > > Usually its a sign that a test is not handling failures
> appropriately,
> > > but
> > > > it is a bit unfortunate that jasmine is so brittle.
> > > >
> > > > -Michal
> > > >
> > > > On Thu, Oct 30, 2014 at 6:10 PM, Toda, Shingo <
> > > ShingoT@fast.au.fujitsu.com
> > > > >
> > > > wrote:
> > > >
> > > > > This is exactly what I got too.
> > > > >
> > > > > Regards,
> > > > > Shingo
> > > > >
> > > > > From: Shazron [mailto:shazron@gmail.com]
> > > > > Sent: Friday, October 31, 2014 9:02 AM
> > > > > To: dev@cordova.apache.org
> > > > > Cc: Toda, Shingo
> > > > > Subject: Re: [mobile-spec] consistency and reliability
> > > > >
> > > > > Seems to be exactly what I am seeing.
> > > > > Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
> > > > >
> > > > > On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <
> > > > ShingoT@fast.au.fujitsu.com
> > > > > <ma...@fast.au.fujitsu.com>> wrote:
> > > > > Hi Shazron
> > > > >
> > > > > Our team are running mobilespec for Android, iOS and Windows. I
> > cannot
> > > > see
> > > > > the attachment but we also have been seeing what you’ve gotten
> since
> > > new
> > > > > style mobilespec was introduced. It seems that each plugin test run
> > > works
> > > > > fine but if all tests get run at once I am likely to get more than
> > one
> > > > > report in a run reporting that there are a number of specs like 900
> > > specs
> > > > > (I don’t think specs are so much).
> > > > >
> > > > > I noticed that some of the specs were called more than once when I
> > got
> > > > > multiple reports. I usually disable FileTransfer, Globalization and
> > > Media
> > > > > specs at first then run test for them individually because I saw
> some
> > > of
> > > > > those tests got called more than once. This could be just my case
> so
> > it
> > > > > might not be yours.
> > > > >
> > > > > Regards,
> > > > > Shingo
> > > > >
> > > > > From: Shazron [mailto:shazron@gmail.com<ma...@gmail.com>]
> > > > > Sent: Friday, October 31, 2014 7:54 AM
> > > > > To: dev@cordova.apache.org<ma...@cordova.apache.org>
> > > > > Subject: [mobile-spec] consistency and reliability
> > > > >
> > > > >
> > > > > `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios
> > > > > --globalplugins`
> > > > >
> > > > > Has anyone gotten mobile-spec to work consistently? I get wildly
> > > > different
> > > > > results on each run. Sometimes there is more than one report in a
> > run.
> > > > See:
> > > > >
> > > > > [Inline image 1]
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [mobile-spec] consistency and reliability
Posted by Brian LeRoux <b...@brian.io>.
- ya sorry for being confusing TAP is a protocol, that part is neat but I
don't really care
- tap/tape are libs for testing that emit TAP results: this is nice
- tap/tape are NOT global runners so you have to explicitly call them and
load them…this is really nice
the problem w/ jasmine, or one of them anyhow, is the global nature and
magic injection of things
On Mon, Nov 3, 2014 at 1:09 PM, Michal Mocny <mm...@chromium.org> wrote:
> I thought tap was just a format for reporting test results? I think the
> problem is that we have some poorly written tests and the framework is not
> recovering well. Any reporting format is orthogonal (unless perhaps the
> bug is specific to the jasmine-html reporter, but I'm not sure there is any
> evidence of that).
>
> That said, writing well behaving async tests can be difficult, so I would
> hope any test framework would do better to defend against that. Perhaps
> tape is better. During the big test migration I evaluated mocha (BDD
> style) as a replacement for jasmine-2.0 since the test definition format is
> really similar, but it had a silly limitation that made porting some tests
> difficult: no support for async describe() blocks, and could not nest it()
> blocks as a workaround. I don't recall the specific plugin(s) which had an
> issue with this.
>
> -Michal
>
> On Mon, Nov 3, 2014 at 3:05 PM, Marcel Kinard <cm...@gmail.com> wrote:
>
> > I haven't seen tap/tape until Brian mentioned it here. From a quick read,
> > it looks like a port of the tests from Jasmine to tape will be required
> on
> > the runner side. Then we can use whatever reporting we want that can
> > consume tap. So if I understand it correctly, the existing tests aren't
> > throwaway, but getting off Jasmine certainly won't be free. Assuming that
> > tape is less brittle and not substantially less functional that Jasmine,
> > could be a net win. Doing the port looks like it will take a fair amount
> of
> > crank-turning.
> >
> > On Oct 30, 2014, at 8:09 PM, Brian LeRoux <b...@brian.io> wrote:
> >
> > > Nope.
> > >
> > > On Thu, Oct 30, 2014, 5:04 PM Jesse <pu...@gmail.com> wrote:
> > >
> > >> I would much rather we fix things, than continually rewrite + discard,
> > >> which seems to be the norm these days.
> > >> tape/tap would require us to throw away thousands of jasmine2 based
> > tests
> > >> wouldn't it?
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
> > For additional commands, e-mail: dev-help@cordova.apache.org
> >
> >
>
Re: [mobile-spec] consistency and reliability
Posted by Michal Mocny <mm...@chromium.org>.
I thought tap was just a format for reporting test results? I think the
problem is that we have some poorly written tests and the framework is not
recovering well. Any reporting format is orthogonal (unless perhaps the
bug is specific to the jasmine-html reporter, but I'm not sure there is any
evidence of that).
That said, writing well behaving async tests can be difficult, so I would
hope any test framework would do better to defend against that. Perhaps
tape is better. During the big test migration I evaluated mocha (BDD
style) as a replacement for jasmine-2.0 since the test definition format is
really similar, but it had a silly limitation that made porting some tests
difficult: no support for async describe() blocks, and could not nest it()
blocks as a workaround. I don't recall the specific plugin(s) which had an
issue with this.
-Michal
On Mon, Nov 3, 2014 at 3:05 PM, Marcel Kinard <cm...@gmail.com> wrote:
> I haven't seen tap/tape until Brian mentioned it here. From a quick read,
> it looks like a port of the tests from Jasmine to tape will be required on
> the runner side. Then we can use whatever reporting we want that can
> consume tap. So if I understand it correctly, the existing tests aren't
> throwaway, but getting off Jasmine certainly won't be free. Assuming that
> tape is less brittle and not substantially less functional that Jasmine,
> could be a net win. Doing the port looks like it will take a fair amount of
> crank-turning.
>
> On Oct 30, 2014, at 8:09 PM, Brian LeRoux <b...@brian.io> wrote:
>
> > Nope.
> >
> > On Thu, Oct 30, 2014, 5:04 PM Jesse <pu...@gmail.com> wrote:
> >
> >> I would much rather we fix things, than continually rewrite + discard,
> >> which seems to be the norm these days.
> >> tape/tap would require us to throw away thousands of jasmine2 based
> tests
> >> wouldn't it?
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
> For additional commands, e-mail: dev-help@cordova.apache.org
>
>
Re: [mobile-spec] consistency and reliability
Posted by Marcel Kinard <cm...@gmail.com>.
I haven't seen tap/tape until Brian mentioned it here. From a quick read, it looks like a port of the tests from Jasmine to tape will be required on the runner side. Then we can use whatever reporting we want that can consume tap. So if I understand it correctly, the existing tests aren't throwaway, but getting off Jasmine certainly won't be free. Assuming that tape is less brittle and not substantially less functional that Jasmine, could be a net win. Doing the port looks like it will take a fair amount of crank-turning.
On Oct 30, 2014, at 8:09 PM, Brian LeRoux <b...@brian.io> wrote:
> Nope.
>
> On Thu, Oct 30, 2014, 5:04 PM Jesse <pu...@gmail.com> wrote:
>
>> I would much rather we fix things, than continually rewrite + discard,
>> which seems to be the norm these days.
>> tape/tap would require us to throw away thousands of jasmine2 based tests
>> wouldn't it?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cordova.apache.org
For additional commands, e-mail: dev-help@cordova.apache.org
RE: [mobile-spec] consistency and reliability
Posted by "Dmitriy Barkalov (Akvelon)" <v-...@microsoft.com>.
Hi Everyone,
It is true that mobile spec test results for Windows are currently not reliable. But our experience allows us to understand which automated tests are expected to fail on Windows / WP8 platforms and which are true fails.
At the same time, we are fixing plugin automated tests for Windows platform and mark unsupported feature tests as pending (e.g. https://github.com/apache/cordova-plugin-contacts/pull/49, https://github.com/apache/cordova-plugin-file/pull/88). We have the same plans for WP8 platform.
To ensure Cordova plugins quality on Windows, we also make complete run for manual plugin tests in addition to mobile spec tests.
Regards, Dmitriy
-----Original Message-----
From: Toda, Shingo [mailto:ShingoT@fast.au.fujitsu.com]
Sent: Friday, October 31, 2014 4:57 AM
To: dev@cordova.apache.org
Cc: Toda, Shingo
Subject: RE: [mobile-spec] consistency and reliability
It's great that this topic has been raised as we have some questions on mobile spec.
We need to test Cordova and core plugins to assure ourselves of the quality so we are running mobile spec and add tests if there are APIs for which we couldn't find tests in mobile spec but I'm interested in how you test Cordova on various platforms.
On Android and iOS, mobile spec runs nicely but on Windows 8 mobile spec there are many crashes due to unsupported API, alert undefined, file paths etc. I am wondering how you test Cordova for Windows 8.
If you don't mind, I would like to know if you are using (new/old) mobilespec or other test suite for testing Windows 8.
Is there any strategy for testing platform-specific APIs which not all platforms support?
Regards,
Shingo
-----Original Message-----
From: Brian LeRoux [mailto:b@brian.io]
Sent: Friday, October 31, 2014 11:09 AM
To: dev@cordova.apache.org
Subject: Re: [mobile-spec] consistency and reliability
Nope.
On Thu, Oct 30, 2014, 5:04 PM Jesse <pu...@gmail.com> wrote:
> I would much rather we fix things, than continually rewrite + discard,
> which seems to be the norm these days.
> tape/tap would require us to throw away thousands of jasmine2 based
> tests wouldn't it?
>
>
> @purplecabbage
> risingj.com
>
> On Thu, Oct 30, 2014 at 4:44 PM, Brian LeRoux <b...@brian.io> wrote:
>
> > tape (and or tap) has no globals and simple / shallow callback
> > solution which fixes that
> >
> > we will have fresh intern blood to sacrifice to the test lib gods
> > alter soon..
> >
> > On Thu, Oct 30, 2014, 4:10 PM Michal Mocny <mm...@chromium.org> wrote:
> >
> > > I've seen that in the past and it usually happens due to misuse of
> > jasmine,
> > > which seems unfortunately quite brittle. Specifically, if done()
> > > isn't called, or an async event fires after a test has already
> > > failed / timed
> > out
> > > and calls done(), or registers new tests with it() etc, weird
> > > things
> will
> > > happen.
> > >
> > > Usually its a sign that a test is not handling failures
> > > appropriately,
> > but
> > > it is a bit unfortunate that jasmine is so brittle.
> > >
> > > -Michal
> > >
> > > On Thu, Oct 30, 2014 at 6:10 PM, Toda, Shingo <
> > ShingoT@fast.au.fujitsu.com
> > > >
> > > wrote:
> > >
> > > > This is exactly what I got too.
> > > >
> > > > Regards,
> > > > Shingo
> > > >
> > > > From: Shazron [mailto:shazron@gmail.com]
> > > > Sent: Friday, October 31, 2014 9:02 AM
> > > > To: dev@cordova.apache.org
> > > > Cc: Toda, Shingo
> > > > Subject: Re: [mobile-spec] consistency and reliability
> > > >
> > > > Seems to be exactly what I am seeing.
> > > > Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
> > > >
> > > > On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <
> > > ShingoT@fast.au.fujitsu.com
> > > > <ma...@fast.au.fujitsu.com>> wrote:
> > > > Hi Shazron
> > > >
> > > > Our team are running mobilespec for Android, iOS and Windows. I
> cannot
> > > see
> > > > the attachment but we also have been seeing what you’ve gotten
> > > > since
> > new
> > > > style mobilespec was introduced. It seems that each plugin test
> > > > run
> > works
> > > > fine but if all tests get run at once I am likely to get more
> > > > than
> one
> > > > report in a run reporting that there are a number of specs like
> > > > 900
> > specs
> > > > (I don’t think specs are so much).
> > > >
> > > > I noticed that some of the specs were called more than once when
> > > > I
> got
> > > > multiple reports. I usually disable FileTransfer, Globalization
> > > > and
> > Media
> > > > specs at first then run test for them individually because I saw
> > > > some
> > of
> > > > those tests got called more than once. This could be just my
> > > > case so
> it
> > > > might not be yours.
> > > >
> > > > Regards,
> > > > Shingo
> > > >
> > > > From: Shazron
> > > > [mailto:shazron@gmail.com<ma...@gmail.com>]
> > > > Sent: Friday, October 31, 2014 7:54 AM
> > > > To: dev@cordova.apache.org<ma...@cordova.apache.org>
> > > > Subject: [mobile-spec] consistency and reliability
> > > >
> > > >
> > > > `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios
> > > > --globalplugins`
> > > >
> > > > Has anyone gotten mobile-spec to work consistently? I get wildly
> > > different
> > > > results on each run. Sometimes there is more than one report in
> > > > a
> run.
> > > See:
> > > >
> > > > [Inline image 1]
> > > >
> > > >
> > >
> >
>
RE: [mobile-spec] consistency and reliability
Posted by "Toda, Shingo" <Sh...@fast.au.fujitsu.com>.
It's great that this topic has been raised as we have some questions on mobile spec.
We need to test Cordova and core plugins to assure ourselves of the quality so we are running mobile spec and add tests if there are APIs for which we couldn't find tests in mobile spec but I'm interested in how you test Cordova on various platforms.
On Android and iOS, mobile spec runs nicely but on Windows 8 mobile spec there are many crashes due to unsupported API, alert undefined, file paths etc. I am wondering how you test Cordova for Windows 8.
If you don't mind, I would like to know if you are using (new/old) mobilespec or other test suite for testing Windows 8.
Is there any strategy for testing platform-specific APIs which not all platforms support?
Regards,
Shingo
-----Original Message-----
From: Brian LeRoux [mailto:b@brian.io]
Sent: Friday, October 31, 2014 11:09 AM
To: dev@cordova.apache.org
Subject: Re: [mobile-spec] consistency and reliability
Nope.
On Thu, Oct 30, 2014, 5:04 PM Jesse <pu...@gmail.com> wrote:
> I would much rather we fix things, than continually rewrite + discard,
> which seems to be the norm these days.
> tape/tap would require us to throw away thousands of jasmine2 based
> tests wouldn't it?
>
>
> @purplecabbage
> risingj.com
>
> On Thu, Oct 30, 2014 at 4:44 PM, Brian LeRoux <b...@brian.io> wrote:
>
> > tape (and or tap) has no globals and simple / shallow callback
> > solution which fixes that
> >
> > we will have fresh intern blood to sacrifice to the test lib gods
> > alter soon..
> >
> > On Thu, Oct 30, 2014, 4:10 PM Michal Mocny <mm...@chromium.org> wrote:
> >
> > > I've seen that in the past and it usually happens due to misuse of
> > jasmine,
> > > which seems unfortunately quite brittle. Specifically, if done()
> > > isn't called, or an async event fires after a test has already
> > > failed / timed
> > out
> > > and calls done(), or registers new tests with it() etc, weird
> > > things
> will
> > > happen.
> > >
> > > Usually its a sign that a test is not handling failures
> > > appropriately,
> > but
> > > it is a bit unfortunate that jasmine is so brittle.
> > >
> > > -Michal
> > >
> > > On Thu, Oct 30, 2014 at 6:10 PM, Toda, Shingo <
> > ShingoT@fast.au.fujitsu.com
> > > >
> > > wrote:
> > >
> > > > This is exactly what I got too.
> > > >
> > > > Regards,
> > > > Shingo
> > > >
> > > > From: Shazron [mailto:shazron@gmail.com]
> > > > Sent: Friday, October 31, 2014 9:02 AM
> > > > To: dev@cordova.apache.org
> > > > Cc: Toda, Shingo
> > > > Subject: Re: [mobile-spec] consistency and reliability
> > > >
> > > > Seems to be exactly what I am seeing.
> > > > Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
> > > >
> > > > On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <
> > > ShingoT@fast.au.fujitsu.com
> > > > <ma...@fast.au.fujitsu.com>> wrote:
> > > > Hi Shazron
> > > >
> > > > Our team are running mobilespec for Android, iOS and Windows. I
> cannot
> > > see
> > > > the attachment but we also have been seeing what you’ve gotten
> > > > since
> > new
> > > > style mobilespec was introduced. It seems that each plugin test
> > > > run
> > works
> > > > fine but if all tests get run at once I am likely to get more
> > > > than
> one
> > > > report in a run reporting that there are a number of specs like
> > > > 900
> > specs
> > > > (I don’t think specs are so much).
> > > >
> > > > I noticed that some of the specs were called more than once when
> > > > I
> got
> > > > multiple reports. I usually disable FileTransfer, Globalization
> > > > and
> > Media
> > > > specs at first then run test for them individually because I saw
> > > > some
> > of
> > > > those tests got called more than once. This could be just my
> > > > case so
> it
> > > > might not be yours.
> > > >
> > > > Regards,
> > > > Shingo
> > > >
> > > > From: Shazron
> > > > [mailto:shazron@gmail.com<ma...@gmail.com>]
> > > > Sent: Friday, October 31, 2014 7:54 AM
> > > > To: dev@cordova.apache.org<ma...@cordova.apache.org>
> > > > Subject: [mobile-spec] consistency and reliability
> > > >
> > > >
> > > > `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios
> > > > --globalplugins`
> > > >
> > > > Has anyone gotten mobile-spec to work consistently? I get wildly
> > > different
> > > > results on each run. Sometimes there is more than one report in
> > > > a
> run.
> > > See:
> > > >
> > > > [Inline image 1]
> > > >
> > > >
> > >
> >
>
Re: [mobile-spec] consistency and reliability
Posted by Brian LeRoux <b...@brian.io>.
Nope.
On Thu, Oct 30, 2014, 5:04 PM Jesse <pu...@gmail.com> wrote:
> I would much rather we fix things, than continually rewrite + discard,
> which seems to be the norm these days.
> tape/tap would require us to throw away thousands of jasmine2 based tests
> wouldn't it?
>
>
> @purplecabbage
> risingj.com
>
> On Thu, Oct 30, 2014 at 4:44 PM, Brian LeRoux <b...@brian.io> wrote:
>
> > tape (and or tap) has no globals and simple / shallow callback solution
> > which fixes that
> >
> > we will have fresh intern blood to sacrifice to the test lib gods alter
> > soon..
> >
> > On Thu, Oct 30, 2014, 4:10 PM Michal Mocny <mm...@chromium.org> wrote:
> >
> > > I've seen that in the past and it usually happens due to misuse of
> > jasmine,
> > > which seems unfortunately quite brittle. Specifically, if done() isn't
> > > called, or an async event fires after a test has already failed / timed
> > out
> > > and calls done(), or registers new tests with it() etc, weird things
> will
> > > happen.
> > >
> > > Usually its a sign that a test is not handling failures appropriately,
> > but
> > > it is a bit unfortunate that jasmine is so brittle.
> > >
> > > -Michal
> > >
> > > On Thu, Oct 30, 2014 at 6:10 PM, Toda, Shingo <
> > ShingoT@fast.au.fujitsu.com
> > > >
> > > wrote:
> > >
> > > > This is exactly what I got too.
> > > >
> > > > Regards,
> > > > Shingo
> > > >
> > > > From: Shazron [mailto:shazron@gmail.com]
> > > > Sent: Friday, October 31, 2014 9:02 AM
> > > > To: dev@cordova.apache.org
> > > > Cc: Toda, Shingo
> > > > Subject: Re: [mobile-spec] consistency and reliability
> > > >
> > > > Seems to be exactly what I am seeing.
> > > > Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
> > > >
> > > > On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <
> > > ShingoT@fast.au.fujitsu.com
> > > > <ma...@fast.au.fujitsu.com>> wrote:
> > > > Hi Shazron
> > > >
> > > > Our team are running mobilespec for Android, iOS and Windows. I
> cannot
> > > see
> > > > the attachment but we also have been seeing what you’ve gotten since
> > new
> > > > style mobilespec was introduced. It seems that each plugin test run
> > works
> > > > fine but if all tests get run at once I am likely to get more than
> one
> > > > report in a run reporting that there are a number of specs like 900
> > specs
> > > > (I don’t think specs are so much).
> > > >
> > > > I noticed that some of the specs were called more than once when I
> got
> > > > multiple reports. I usually disable FileTransfer, Globalization and
> > Media
> > > > specs at first then run test for them individually because I saw some
> > of
> > > > those tests got called more than once. This could be just my case so
> it
> > > > might not be yours.
> > > >
> > > > Regards,
> > > > Shingo
> > > >
> > > > From: Shazron [mailto:shazron@gmail.com<ma...@gmail.com>]
> > > > Sent: Friday, October 31, 2014 7:54 AM
> > > > To: dev@cordova.apache.org<ma...@cordova.apache.org>
> > > > Subject: [mobile-spec] consistency and reliability
> > > >
> > > >
> > > > `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios
> > > > --globalplugins`
> > > >
> > > > Has anyone gotten mobile-spec to work consistently? I get wildly
> > > different
> > > > results on each run. Sometimes there is more than one report in a
> run.
> > > See:
> > > >
> > > > [Inline image 1]
> > > >
> > > >
> > >
> >
>
Re: [mobile-spec] consistency and reliability
Posted by Jesse <pu...@gmail.com>.
I would much rather we fix things, than continually rewrite + discard,
which seems to be the norm these days.
tape/tap would require us to throw away thousands of jasmine2 based tests
wouldn't it?
@purplecabbage
risingj.com
On Thu, Oct 30, 2014 at 4:44 PM, Brian LeRoux <b...@brian.io> wrote:
> tape (and or tap) has no globals and simple / shallow callback solution
> which fixes that
>
> we will have fresh intern blood to sacrifice to the test lib gods alter
> soon..
>
> On Thu, Oct 30, 2014, 4:10 PM Michal Mocny <mm...@chromium.org> wrote:
>
> > I've seen that in the past and it usually happens due to misuse of
> jasmine,
> > which seems unfortunately quite brittle. Specifically, if done() isn't
> > called, or an async event fires after a test has already failed / timed
> out
> > and calls done(), or registers new tests with it() etc, weird things will
> > happen.
> >
> > Usually its a sign that a test is not handling failures appropriately,
> but
> > it is a bit unfortunate that jasmine is so brittle.
> >
> > -Michal
> >
> > On Thu, Oct 30, 2014 at 6:10 PM, Toda, Shingo <
> ShingoT@fast.au.fujitsu.com
> > >
> > wrote:
> >
> > > This is exactly what I got too.
> > >
> > > Regards,
> > > Shingo
> > >
> > > From: Shazron [mailto:shazron@gmail.com]
> > > Sent: Friday, October 31, 2014 9:02 AM
> > > To: dev@cordova.apache.org
> > > Cc: Toda, Shingo
> > > Subject: Re: [mobile-spec] consistency and reliability
> > >
> > > Seems to be exactly what I am seeing.
> > > Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
> > >
> > > On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <
> > ShingoT@fast.au.fujitsu.com
> > > <ma...@fast.au.fujitsu.com>> wrote:
> > > Hi Shazron
> > >
> > > Our team are running mobilespec for Android, iOS and Windows. I cannot
> > see
> > > the attachment but we also have been seeing what you’ve gotten since
> new
> > > style mobilespec was introduced. It seems that each plugin test run
> works
> > > fine but if all tests get run at once I am likely to get more than one
> > > report in a run reporting that there are a number of specs like 900
> specs
> > > (I don’t think specs are so much).
> > >
> > > I noticed that some of the specs were called more than once when I got
> > > multiple reports. I usually disable FileTransfer, Globalization and
> Media
> > > specs at first then run test for them individually because I saw some
> of
> > > those tests got called more than once. This could be just my case so it
> > > might not be yours.
> > >
> > > Regards,
> > > Shingo
> > >
> > > From: Shazron [mailto:shazron@gmail.com<ma...@gmail.com>]
> > > Sent: Friday, October 31, 2014 7:54 AM
> > > To: dev@cordova.apache.org<ma...@cordova.apache.org>
> > > Subject: [mobile-spec] consistency and reliability
> > >
> > >
> > > `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios
> > > --globalplugins`
> > >
> > > Has anyone gotten mobile-spec to work consistently? I get wildly
> > different
> > > results on each run. Sometimes there is more than one report in a run.
> > See:
> > >
> > > [Inline image 1]
> > >
> > >
> >
>
Re: [mobile-spec] consistency and reliability
Posted by Brian LeRoux <b...@brian.io>.
tape (and or tap) has no globals and simple / shallow callback solution
which fixes that
we will have fresh intern blood to sacrifice to the test lib gods alter
soon..
On Thu, Oct 30, 2014, 4:10 PM Michal Mocny <mm...@chromium.org> wrote:
> I've seen that in the past and it usually happens due to misuse of jasmine,
> which seems unfortunately quite brittle. Specifically, if done() isn't
> called, or an async event fires after a test has already failed / timed out
> and calls done(), or registers new tests with it() etc, weird things will
> happen.
>
> Usually its a sign that a test is not handling failures appropriately, but
> it is a bit unfortunate that jasmine is so brittle.
>
> -Michal
>
> On Thu, Oct 30, 2014 at 6:10 PM, Toda, Shingo <ShingoT@fast.au.fujitsu.com
> >
> wrote:
>
> > This is exactly what I got too.
> >
> > Regards,
> > Shingo
> >
> > From: Shazron [mailto:shazron@gmail.com]
> > Sent: Friday, October 31, 2014 9:02 AM
> > To: dev@cordova.apache.org
> > Cc: Toda, Shingo
> > Subject: Re: [mobile-spec] consistency and reliability
> >
> > Seems to be exactly what I am seeing.
> > Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
> >
> > On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <
> ShingoT@fast.au.fujitsu.com
> > <ma...@fast.au.fujitsu.com>> wrote:
> > Hi Shazron
> >
> > Our team are running mobilespec for Android, iOS and Windows. I cannot
> see
> > the attachment but we also have been seeing what you’ve gotten since new
> > style mobilespec was introduced. It seems that each plugin test run works
> > fine but if all tests get run at once I am likely to get more than one
> > report in a run reporting that there are a number of specs like 900 specs
> > (I don’t think specs are so much).
> >
> > I noticed that some of the specs were called more than once when I got
> > multiple reports. I usually disable FileTransfer, Globalization and Media
> > specs at first then run test for them individually because I saw some of
> > those tests got called more than once. This could be just my case so it
> > might not be yours.
> >
> > Regards,
> > Shingo
> >
> > From: Shazron [mailto:shazron@gmail.com<ma...@gmail.com>]
> > Sent: Friday, October 31, 2014 7:54 AM
> > To: dev@cordova.apache.org<ma...@cordova.apache.org>
> > Subject: [mobile-spec] consistency and reliability
> >
> >
> > `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios
> > --globalplugins`
> >
> > Has anyone gotten mobile-spec to work consistently? I get wildly
> different
> > results on each run. Sometimes there is more than one report in a run.
> See:
> >
> > [Inline image 1]
> >
> >
>
Re: [mobile-spec] consistency and reliability
Posted by Michal Mocny <mm...@chromium.org>.
I've seen that in the past and it usually happens due to misuse of jasmine,
which seems unfortunately quite brittle. Specifically, if done() isn't
called, or an async event fires after a test has already failed / timed out
and calls done(), or registers new tests with it() etc, weird things will
happen.
Usually its a sign that a test is not handling failures appropriately, but
it is a bit unfortunate that jasmine is so brittle.
-Michal
On Thu, Oct 30, 2014 at 6:10 PM, Toda, Shingo <Sh...@fast.au.fujitsu.com>
wrote:
> This is exactly what I got too.
>
> Regards,
> Shingo
>
> From: Shazron [mailto:shazron@gmail.com]
> Sent: Friday, October 31, 2014 9:02 AM
> To: dev@cordova.apache.org
> Cc: Toda, Shingo
> Subject: Re: [mobile-spec] consistency and reliability
>
> Seems to be exactly what I am seeing.
> Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
>
> On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <ShingoT@fast.au.fujitsu.com
> <ma...@fast.au.fujitsu.com>> wrote:
> Hi Shazron
>
> Our team are running mobilespec for Android, iOS and Windows. I cannot see
> the attachment but we also have been seeing what you’ve gotten since new
> style mobilespec was introduced. It seems that each plugin test run works
> fine but if all tests get run at once I am likely to get more than one
> report in a run reporting that there are a number of specs like 900 specs
> (I don’t think specs are so much).
>
> I noticed that some of the specs were called more than once when I got
> multiple reports. I usually disable FileTransfer, Globalization and Media
> specs at first then run test for them individually because I saw some of
> those tests got called more than once. This could be just my case so it
> might not be yours.
>
> Regards,
> Shingo
>
> From: Shazron [mailto:shazron@gmail.com<ma...@gmail.com>]
> Sent: Friday, October 31, 2014 7:54 AM
> To: dev@cordova.apache.org<ma...@cordova.apache.org>
> Subject: [mobile-spec] consistency and reliability
>
>
> `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios
> --globalplugins`
>
> Has anyone gotten mobile-spec to work consistently? I get wildly different
> results on each run. Sometimes there is more than one report in a run. See:
>
> [Inline image 1]
>
>
RE: [mobile-spec] consistency and reliability
Posted by "Toda, Shingo" <Sh...@fast.au.fujitsu.com>.
This is exactly what I got too.
Regards,
Shingo
From: Shazron [mailto:shazron@gmail.com]
Sent: Friday, October 31, 2014 9:02 AM
To: dev@cordova.apache.org
Cc: Toda, Shingo
Subject: Re: [mobile-spec] consistency and reliability
Seems to be exactly what I am seeing.
Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <Sh...@fast.au.fujitsu.com>> wrote:
Hi Shazron
Our team are running mobilespec for Android, iOS and Windows. I cannot see the attachment but we also have been seeing what you’ve gotten since new style mobilespec was introduced. It seems that each plugin test run works fine but if all tests get run at once I am likely to get more than one report in a run reporting that there are a number of specs like 900 specs (I don’t think specs are so much).
I noticed that some of the specs were called more than once when I got multiple reports. I usually disable FileTransfer, Globalization and Media specs at first then run test for them individually because I saw some of those tests got called more than once. This could be just my case so it might not be yours.
Regards,
Shingo
From: Shazron [mailto:shazron@gmail.com<ma...@gmail.com>]
Sent: Friday, October 31, 2014 7:54 AM
To: dev@cordova.apache.org<ma...@cordova.apache.org>
Subject: [mobile-spec] consistency and reliability
`cordova-mobile-spec/createmobilespec/createmobilespec.js --ios --globalplugins`
Has anyone gotten mobile-spec to work consistently? I get wildly different results on each run. Sometimes there is more than one report in a run. See:
[Inline image 1]
Re: [mobile-spec] consistency and reliability
Posted by Shazron <sh...@gmail.com>.
Seems to be exactly what I am seeing.
Here's the image as a link: http://i.imgur.com/n6ST1Kd.png?1
On Thu, Oct 30, 2014 at 2:57 PM, Toda, Shingo <Sh...@fast.au.fujitsu.com>
wrote:
> Hi Shazron
>
> Our team are running mobilespec for Android, iOS and Windows. I cannot see
> the attachment but we also have been seeing what you’ve gotten since new
> style mobilespec was introduced. It seems that each plugin test run works
> fine but if all tests get run at once I am likely to get more than one
> report in a run reporting that there are a number of specs like 900 specs
> (I don’t think specs are so much).
>
> I noticed that some of the specs were called more than once when I got
> multiple reports. I usually disable FileTransfer, Globalization and Media
> specs at first then run test for them individually because I saw some of
> those tests got called more than once. This could be just my case so it
> might not be yours.
>
> Regards,
> Shingo
>
> From: Shazron [mailto:shazron@gmail.com]
> Sent: Friday, October 31, 2014 7:54 AM
> To: dev@cordova.apache.org
> Subject: [mobile-spec] consistency and reliability
>
>
> `cordova-mobile-spec/createmobilespec/createmobilespec.js --ios
> --globalplugins`
>
> Has anyone gotten mobile-spec to work consistently? I get wildly different
> results on each run. Sometimes there is more than one report in a run. See:
>
> [Inline image 1]
>
RE: [mobile-spec] consistency and reliability
Posted by "Toda, Shingo" <Sh...@fast.au.fujitsu.com>.
Hi Shazron
Our team are running mobilespec for Android, iOS and Windows. I cannot see the attachment but we also have been seeing what you’ve gotten since new style mobilespec was introduced. It seems that each plugin test run works fine but if all tests get run at once I am likely to get more than one report in a run reporting that there are a number of specs like 900 specs (I don’t think specs are so much).
I noticed that some of the specs were called more than once when I got multiple reports. I usually disable FileTransfer, Globalization and Media specs at first then run test for them individually because I saw some of those tests got called more than once. This could be just my case so it might not be yours.
Regards,
Shingo
From: Shazron [mailto:shazron@gmail.com]
Sent: Friday, October 31, 2014 7:54 AM
To: dev@cordova.apache.org
Subject: [mobile-spec] consistency and reliability
`cordova-mobile-spec/createmobilespec/createmobilespec.js --ios --globalplugins`
Has anyone gotten mobile-spec to work consistently? I get wildly different results on each run. Sometimes there is more than one report in a run. See:
[Inline image 1]