Release testing gaps

[Nikhil Khandelwal <ni...@microsoft.com>]

https://issues.apache.org/jira/browse/CB-9248 highlights a testing gap with our release process:

*         cordova-plugin-whitelist should not have been released with having iOS platform >= 4.0 without iOS platform release. While reverting the 'latest' tag was a good move to get this fixed, it would have been nice to catch it earlier.

*         cordova-plugin-file-transfer should not have been shipped with a build failure on WP8 because it references unreleased types in WP8 platform in master.
While we are using buildbot to run and the build the plugin against master version of the platform, most users are using it with the released version of the platform. At the very least, this scenario needs to be tested at the time of release, if not in a CI scenario with every commit.

Would love to hear your thoughts on how we can catch these issues before release? One of the ideas is to use buildbot to schedule a 'release' build that would run the tests we have with the correct versions of the code that we want - release tag for plugins against released versions of the platform.

Thanks,
Nikhil

Re: Release testing gaps

[Carlos Santana <cs...@gmail.com>]

yep, I agree with Steven to test the potential combinations to be released

" It would be
nice to test master plugins against released platforms and master platforms
with released plugins."


On Mon, Jul 6, 2015 at 2:26 PM Steven Gill <st...@gmail.com> wrote:

> Do we currently just test master plugins with master platforms? It would be
> nice to test master plugins against released platforms and master platforms
> with released plugins.
>
> cordova-plugin-whitelist problem got solved in a unreleased fix to
> cordova-lib. It would have made sense to hold off on releasing this plugin
> until we did a tools release. I reverted the latest tag when this issue
> came up on the list.
>
> I wish I had caught the cordova-plugin-file-transfer bug. It was showing
> those tests failing over a few builds which made me think the fails were
> expected. Maybe we should have a expected fails per platform/plugin chart
> of some sort to quickly compare against?
>
> -Steve
>
>
>
> On Mon, Jul 6, 2015 at 10:16 AM, Nikhil Khandelwal <nikhilkh@microsoft.com
> >
> wrote:
>
> > https://issues.apache.org/jira/browse/CB-9248 highlights a testing gap
> > with our release process:
> >
> > *         cordova-plugin-whitelist should not have been released with
> > having iOS platform >= 4.0 without iOS platform release. While reverting
> > the 'latest' tag was a good move to get this fixed, it would have been
> nice
> > to catch it earlier.
> >
> > *         cordova-plugin-file-transfer should not have been shipped with
> a
> > build failure on WP8 because it references unreleased types in WP8
> platform
> > in master.
> > While we are using buildbot to run and the build the plugin against
> master
> > version of the platform, most users are using it with the released
> version
> > of the platform. At the very least, this scenario needs to be tested at
> the
> > time of release, if not in a CI scenario with every commit.
> >
> > Would love to hear your thoughts on how we can catch these issues before
> > release? One of the ideas is to use buildbot to schedule a 'release'
> build
> > that would run the tests we have with the correct versions of the code
> that
> > we want - release tag for plugins against released versions of the
> platform.
> >
> > Thanks,
> > Nikhil
> >
> >
> >
>

Re: Release testing gaps

[Steven Gill <st...@gmail.com>]

Do we currently just test master plugins with master platforms? It would be
nice to test master plugins against released platforms and master platforms
with released plugins.

cordova-plugin-whitelist problem got solved in a unreleased fix to
cordova-lib. It would have made sense to hold off on releasing this plugin
until we did a tools release. I reverted the latest tag when this issue
came up on the list.

I wish I had caught the cordova-plugin-file-transfer bug. It was showing
those tests failing over a few builds which made me think the fails were
expected. Maybe we should have a expected fails per platform/plugin chart
of some sort to quickly compare against?

-Steve



On Mon, Jul 6, 2015 at 10:16 AM, Nikhil Khandelwal <ni...@microsoft.com>
wrote:

> https://issues.apache.org/jira/browse/CB-9248 highlights a testing gap
> with our release process:
>
> *         cordova-plugin-whitelist should not have been released with
> having iOS platform >= 4.0 without iOS platform release. While reverting
> the 'latest' tag was a good move to get this fixed, it would have been nice
> to catch it earlier.
>
> *         cordova-plugin-file-transfer should not have been shipped with a
> build failure on WP8 because it references unreleased types in WP8 platform
> in master.
> While we are using buildbot to run and the build the plugin against master
> version of the platform, most users are using it with the released version
> of the platform. At the very least, this scenario needs to be tested at the
> time of release, if not in a CI scenario with every commit.
>
> Would love to hear your thoughts on how we can catch these issues before
> release? One of the ideas is to use buildbot to schedule a 'release' build
> that would run the tests we have with the correct versions of the code that
> we want - release tag for plugins against released versions of the platform.
>
> Thanks,
> Nikhil
>
>
>