collecting mobile-spec test results and generating reports

[Marcel Kinard <cm...@gmail.com>]

Here is something I feel is important and I wanted to see what you think.

Problem definition: Cordova is a bit unique in that it not only has to 
deal with running on a large number of permutations of device models and 
device OS's, but the testing is started manually by a large number of 
people using devices in their desk drawer, it is very decentralized. My 
fear is that there are invisible holes and dups in our testing at the 
end of a release. Thus we are releasing code for environments we didn't 
test, but perhaps could have tested. And are we really running both the 
automatic and manual tests in mobile-spec, or just the automatic tests? 
In a more generalized perspective, we don't have good visibility to what 
environments on which tests ran, and visibility to test failures as we 
evaluate release readiness, other than a few comments on the dev mailing 
list.

Proposed solution: mobile-spec is a reasonably good test suite. But the 
test results never leave the device. So the suggestion is to add a 
"Submit test results" button throughout mobile-spec (at each place where 
test results are generated) that would be an opt-in mechanism to get the 
pass/fail test results off the device. Included in the submission with 
the test results would be a bit of metadata, such as the device make and 
model and OS version, Cordova version, etc. The submission would get 
posted to a centralized db somewhere. A web-based query could be run 
against the db to list the submitted results, with the intention to 
understand which environments have been tested for a particular Cordova 
version, which tests ran, and which tests failed/succeeded.

It should be opt-in instead of opt-out because we generally don't want 
test results from a developer that is working on a new feature in the 
middle of a dev cycle. The intent here is to capture tests from rc's and 
similar as we approach the end of a release. It's about evaluating 
release readiness. Maybe in the long term we want to evaluate release 
readiness all the way through a release, but that's not the short-term 
intention here.

I love the vision of continuous integration test, and see this proposal 
as a first step towards that.

Start shooting holes: Reply with what you think. Before we dive into 
where the db should be hosted and a wire protocol for the submission, at 
a high level do you agree with the problem definition and do you have 
any other ideas/comments on a solution? And what do you feel is the 
urgency for this? Thanks!

-- Marcel Kinard

Re: collecting mobile-spec test results and generating reports

[Michael Brooks <mi...@michaelbrooks.ca>]

Great discussion point Marcel!

As Fil has mentioned, we've started some of this work and it aligns very
closely with your described solution.

At the moment, Medic is wired up to our device wall, but I see it extending
well beyond that. I would love to see the day where Apache Cordova
committers can `npm install -g cordova-medic` and run medic locally. As
commits are made to the Apache Cordova projects, your local devices will
automatically run mobile-spec against the latest changes and report back to
the CouchDB instance. In other words, a de-centralized continuous
integration service.

Michael

On Thu, Jan 10, 2013 at 9:58 AM, Filip Maj <fi...@adobe.com> wrote:

> Hey Marcel, I am working towards that continuous integration vision.
>
> github.com/filmaj/medic
>
> It is a node process that builds the latest commits coming into android,
> iOS + BlackBerry repos, pairs them with mobile-spec, scans for
> connected/nearby devices, and deploys the mobile-spec app to those
> devices. It modifies mobile-spec so that all test results are sent off to
> a couchdb server. There is a "dashboard" script you can run that is a
> little node server that aggregates test results from this couchdb server
> and displays. The dashboard is also live here:
>
> http://cordova.medic.jit.su/
>
>
> Currently the Adobe Vancouver office device wall has this setup running on
> it (actually, not at this EXACT time but hopefully be back online by
> tomorrow :) ). Very recently the gitpubsub service over at Apache went
> live (try running curl http://urd.zones.apache.org:2069/json) which
> broadcasts all git commits going into all apache git repos. I am in the
> process of wiring up medic to that new service to pick out cordova commits
> as-they-happen.
>
> The idea is, at least now, that this is a setup that we can distribute to
> almost anyone and have them submit tests to our centralized server. So, a
> mobile, distributed CI setup, if you will. It's a lot of fun to play with!
>
> The couchdb server at this point is an Adobe EC2 instance. I haven't set
> up any special permissions (I think it's still in Admin Party mode) so if
> we were to publicize that server I'd have to set it up so that, at the
> minimum, no one can delete test results :)
>
> Overall, though, I like your idea of the opt-in submission button. Would
> love to see that in mobile spec. Also I'd love feedback on medic!
>
> Cheers,
> Fil
>
> On 1/10/13 8:30 AM, "Marcel Kinard" <cm...@gmail.com> wrote:
>
> >Here is something I feel is important and I wanted to see what you think.
> >
> >Problem definition: Cordova is a bit unique in that it not only has to
> >deal with running on a large number of permutations of device models and
> >device OS's, but the testing is started manually by a large number of
> >people using devices in their desk drawer, it is very decentralized. My
> >fear is that there are invisible holes and dups in our testing at the
> >end of a release. Thus we are releasing code for environments we didn't
> >test, but perhaps could have tested. And are we really running both the
> >automatic and manual tests in mobile-spec, or just the automatic tests?
> >In a more generalized perspective, we don't have good visibility to what
> >environments on which tests ran, and visibility to test failures as we
> >evaluate release readiness, other than a few comments on the dev mailing
> >list.
> >
> >Proposed solution: mobile-spec is a reasonably good test suite. But the
> >test results never leave the device. So the suggestion is to add a
> >"Submit test results" button throughout mobile-spec (at each place where
> >test results are generated) that would be an opt-in mechanism to get the
> >pass/fail test results off the device. Included in the submission with
> >the test results would be a bit of metadata, such as the device make and
> >model and OS version, Cordova version, etc. The submission would get
> >posted to a centralized db somewhere. A web-based query could be run
> >against the db to list the submitted results, with the intention to
> >understand which environments have been tested for a particular Cordova
> >version, which tests ran, and which tests failed/succeeded.
> >
> >It should be opt-in instead of opt-out because we generally don't want
> >test results from a developer that is working on a new feature in the
> >middle of a dev cycle. The intent here is to capture tests from rc's and
> >similar as we approach the end of a release. It's about evaluating
> >release readiness. Maybe in the long term we want to evaluate release
> >readiness all the way through a release, but that's not the short-term
> >intention here.
> >
> >I love the vision of continuous integration test, and see this proposal
> >as a first step towards that.
> >
> >Start shooting holes: Reply with what you think. Before we dive into
> >where the db should be hosted and a wire protocol for the submission, at
> >a high level do you agree with the problem definition and do you have
> >any other ideas/comments on a solution? And what do you feel is the
> >urgency for this? Thanks!
> >
> >-- Marcel Kinard
>
>

Re: collecting mobile-spec test results and generating reports

[Filip Maj <fi...@adobe.com>]

Take the medic code for a spin and see how it works! If you have questions
don't hesitate to ping me. Feel free to file issues on github as well.
I've demoed it on a few different computers now (and even during a talk)
so its become quite durable over the last 6 months or so that I've been
working on it.

On 1/10/13 2:43 PM, "Marcel Kinard" <cm...@gmail.com> wrote:

>"I gotta get me one of these!" --Tangled (I have kids.)
>
>I wasn't aware this existed. Thank you much for the writeup. Just from
>what I've read here, I will definitely be looking more into this. For
>quite a while we've been contemplating a similar device wall here at IBM.
>
>@michael, yes, a decentralized continuous integration service would be
>nirvana.
>
>-- Marcel Kinard
>
>On 1/10/2013 12:58 PM, Filip Maj wrote:
>> Hey Marcel, I am working towards that continuous integration vision.
>>
>> github.com/filmaj/medic
>>
>> It is a node process that builds the latest commits coming into android,
>> iOS + BlackBerry repos, pairs them with mobile-spec, scans for
>> connected/nearby devices, and deploys the mobile-spec app to those
>> devices. It modifies mobile-spec so that all test results are sent off
>>to
>> a couchdb server. There is a "dashboard" script you can run that is a
>> little node server that aggregates test results from this couchdb server
>> and displays. The dashboard is also live here:
>>
>> http://cordova.medic.jit.su/
>>
>>
>> Currently the Adobe Vancouver office device wall has this setup running
>>on
>> it (actually, not at this EXACT time but hopefully be back online by
>> tomorrow :) ). Very recently the gitpubsub service over at Apache went
>> live (try running curl http://urd.zones.apache.org:2069/json) which
>> broadcasts all git commits going into all apache git repos. I am in the
>> process of wiring up medic to that new service to pick out cordova
>>commits
>> as-they-happen.
>>
>> The idea is, at least now, that this is a setup that we can distribute
>>to
>> almost anyone and have them submit tests to our centralized server. So,
>>a
>> mobile, distributed CI setup, if you will. It's a lot of fun to play
>>with!
>>
>> The couchdb server at this point is an Adobe EC2 instance. I haven't set
>> up any special permissions (I think it's still in Admin Party mode) so
>>if
>> we were to publicize that server I'd have to set it up so that, at the
>> minimum, no one can delete test results :)
>>
>> Overall, though, I like your idea of the opt-in submission button. Would
>> love to see that in mobile spec. Also I'd love feedback on medic!
>>
>> Cheers,
>> Fil
>>
>

Re: collecting mobile-spec test results and generating reports

[Marcel Kinard <cm...@gmail.com>]

"I gotta get me one of these!" --Tangled (I have kids.)

I wasn't aware this existed. Thank you much for the writeup. Just from 
what I've read here, I will definitely be looking more into this. For 
quite a while we've been contemplating a similar device wall here at IBM.

@michael, yes, a decentralized continuous integration service would be 
nirvana.

-- Marcel Kinard

On 1/10/2013 12:58 PM, Filip Maj wrote:
> Hey Marcel, I am working towards that continuous integration vision.
>
> github.com/filmaj/medic
>
> It is a node process that builds the latest commits coming into android,
> iOS + BlackBerry repos, pairs them with mobile-spec, scans for
> connected/nearby devices, and deploys the mobile-spec app to those
> devices. It modifies mobile-spec so that all test results are sent off to
> a couchdb server. There is a "dashboard" script you can run that is a
> little node server that aggregates test results from this couchdb server
> and displays. The dashboard is also live here:
>
> http://cordova.medic.jit.su/
>
>
> Currently the Adobe Vancouver office device wall has this setup running on
> it (actually, not at this EXACT time but hopefully be back online by
> tomorrow :) ). Very recently the gitpubsub service over at Apache went
> live (try running curl http://urd.zones.apache.org:2069/json) which
> broadcasts all git commits going into all apache git repos. I am in the
> process of wiring up medic to that new service to pick out cordova commits
> as-they-happen.
>
> The idea is, at least now, that this is a setup that we can distribute to
> almost anyone and have them submit tests to our centralized server. So, a
> mobile, distributed CI setup, if you will. It's a lot of fun to play with!
>
> The couchdb server at this point is an Adobe EC2 instance. I haven't set
> up any special permissions (I think it's still in Admin Party mode) so if
> we were to publicize that server I'd have to set it up so that, at the
> minimum, no one can delete test results :)
>
> Overall, though, I like your idea of the opt-in submission button. Would
> love to see that in mobile spec. Also I'd love feedback on medic!
>
> Cheers,
> Fil
>

Re: collecting mobile-spec test results and generating reports

[Filip Maj <fi...@adobe.com>]

Hey Marcel, I am working towards that continuous integration vision.

github.com/filmaj/medic

It is a node process that builds the latest commits coming into android,
iOS + BlackBerry repos, pairs them with mobile-spec, scans for
connected/nearby devices, and deploys the mobile-spec app to those
devices. It modifies mobile-spec so that all test results are sent off to
a couchdb server. There is a "dashboard" script you can run that is a
little node server that aggregates test results from this couchdb server
and displays. The dashboard is also live here:

http://cordova.medic.jit.su/


Currently the Adobe Vancouver office device wall has this setup running on
it (actually, not at this EXACT time but hopefully be back online by
tomorrow :) ). Very recently the gitpubsub service over at Apache went
live (try running curl http://urd.zones.apache.org:2069/json) which
broadcasts all git commits going into all apache git repos. I am in the
process of wiring up medic to that new service to pick out cordova commits
as-they-happen.

The idea is, at least now, that this is a setup that we can distribute to
almost anyone and have them submit tests to our centralized server. So, a
mobile, distributed CI setup, if you will. It's a lot of fun to play with!

The couchdb server at this point is an Adobe EC2 instance. I haven't set
up any special permissions (I think it's still in Admin Party mode) so if
we were to publicize that server I'd have to set it up so that, at the
minimum, no one can delete test results :)

Overall, though, I like your idea of the opt-in submission button. Would
love to see that in mobile spec. Also I'd love feedback on medic!

Cheers,
Fil

On 1/10/13 8:30 AM, "Marcel Kinard" <cm...@gmail.com> wrote:

>Here is something I feel is important and I wanted to see what you think.
>
>Problem definition: Cordova is a bit unique in that it not only has to
>deal with running on a large number of permutations of device models and
>device OS's, but the testing is started manually by a large number of
>people using devices in their desk drawer, it is very decentralized. My
>fear is that there are invisible holes and dups in our testing at the
>end of a release. Thus we are releasing code for environments we didn't
>test, but perhaps could have tested. And are we really running both the
>automatic and manual tests in mobile-spec, or just the automatic tests?
>In a more generalized perspective, we don't have good visibility to what
>environments on which tests ran, and visibility to test failures as we
>evaluate release readiness, other than a few comments on the dev mailing
>list.
>
>Proposed solution: mobile-spec is a reasonably good test suite. But the
>test results never leave the device. So the suggestion is to add a
>"Submit test results" button throughout mobile-spec (at each place where
>test results are generated) that would be an opt-in mechanism to get the
>pass/fail test results off the device. Included in the submission with
>the test results would be a bit of metadata, such as the device make and
>model and OS version, Cordova version, etc. The submission would get
>posted to a centralized db somewhere. A web-based query could be run
>against the db to list the submitted results, with the intention to
>understand which environments have been tested for a particular Cordova
>version, which tests ran, and which tests failed/succeeded.
>
>It should be opt-in instead of opt-out because we generally don't want
>test results from a developer that is working on a new feature in the
>middle of a dev cycle. The intent here is to capture tests from rc's and
>similar as we approach the end of a release. It's about evaluating
>release readiness. Maybe in the long term we want to evaluate release
>readiness all the way through a release, but that's not the short-term
>intention here.
>
>I love the vision of continuous integration test, and see this proposal
>as a first step towards that.
>
>Start shooting holes: Reply with what you think. Before we dive into
>where the db should be hosted and a wire protocol for the submission, at
>a high level do you agree with the problem definition and do you have
>any other ideas/comments on a solution? And what do you feel is the
>urgency for this? Thanks!
>
>-- Marcel Kinard