Testing Framework Site Update

[Omar Gonzalez <om...@gmail.com>]

Hello everyone,

As I did not see any more objections to the testing libraries I have
committed the text additions below to the Source page on the project site,
it is not yet published so the changes are not yet visible to the public.

Let me know if you think I should include any changes, I'll be away from my
computer for a few hours but can make edits or publish later tonight.

## Apache Flex SDK Automated Testing Libraries

We use [FlexUnit][4] for unit testing and [Mockolate][5] for mocks and test
spies.
FlexUnit is in the initial phases of being donated to the Apache Flex
project, more
information will be posted here as it becomes available.

Information on Mustella based functional testing will become available here
as soon as
the testing framework has finished going through the donation process from
Adobe
to The Apache Software Foundation.


[4]: http://flexunit.org/
[5]: http://mockolate.org/

--
Omar Gonzalez
s9tpepper@apache.org

Re: Testing Framework Site Update

[Omar Gonzalez <om...@gmail.com>]

>
>
> * For existing code, write tests to verify changes
> * For new code, target 100% coverage
>
> sD
>
>
That's exactly the approach I'm taking with code in my whiteboard area:
http://svn.apache.org/viewvc/incubator/flex/whiteboard/s9tpepper/validators/
.

Of course, that's not to say that if someone wants to sit and write out
unit tests for all the existing framework components that we would not
appreciate it.  Its just not how I want to spend the little time I have to
work on Apache Flex.

--
Omar Gonzalez
s9tpepper@apache.org
Apache Flex PPMC Member

Re: Testing Framework Site Update

[Scott Delamater <sc...@gmail.com>]

Thanks Mike. I saw heaps of discussion around Mustella and regarding the
selection of a test framework, but nothing specifically about the
(non)existence of unit tests for the framework.

I'm more than a little frightened that we don't have test coverage. Without
seeing the tests, I'm guessing that it would be a pretty mammoth effort to
create unit tests for areas not sufficiently covered by Mustella.

I hope this will happen organically (and I realize this could get into an
idealogical debate, which is not my intent), but I suggest that we strongly
encourage a create-as-you-go approach to unit testing, specifically:

* For existing code, write tests to verify changes
* For new code, target 100% coverage

sD

On Tue, Feb 28, 2012 at 7:46 AM, Michael A. Labriola <
labriola@digitalprimates.net> wrote:

> >I assume I'm missing something, but I don't see *any* unit tests in the
> repo. Are we still waiting on those?
>
> Look back through the list, there was a lot of discussion on this. There
> are no unit tests for the Flex framework. There are mustella tests, which
> is proprietary testing framework used by Adobe that will someday be
> contributed along with the tests.
>
> Mike
>
>

RE: Testing Framework Site Update

["Michael A. Labriola" <la...@digitalprimates.net>]

>I assume I'm missing something, but I don't see *any* unit tests in the repo. Are we still waiting on those?

Look back through the list, there was a lot of discussion on this. There are no unit tests for the Flex framework. There are mustella tests, which is proprietary testing framework used by Adobe that will someday be contributed along with the tests.

Mike

Re: Testing Framework Site Update

[Alex Harui <ah...@adobe.com>]

On 2/28/12 10:44 AM, "Michael A. Labriola" <la...@digitalprimates.net>
wrote:

>> My understanding (needs verification) is that we used a tool that matched the
>> output of flash.trace.Trace to the debugline opcodes in a SWF and that is the
>> code >coverage number.  Unless I'm missing something, that is  sufficient,
>> and the example above would not show 100% coverage.
> 
> Thanks Alex, please verify when you can as this is not my understanding. One
> side note, even in the much better case you describe that would still be line
> coverage and I would us to improve our technique there over time.
I received confirmation.  But you are absolutely right that line coverage
doesn't prove out boundary or load handling or other conditions.  That's one
reason why there are still plenty of bugs in the SDK.

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

RE: Testing Framework Site Update

["Michael A. Labriola" <la...@digitalprimates.net>]

>My understanding (needs verification) is that we used a tool that matched the output of flash.trace.Trace to the debugline opcodes in a SWF and that is the code >coverage number.  Unless I'm missing something, that is  sufficient, and the example above would not show 100% coverage.

Thanks Alex, please verify when you can as this is not my understanding. One side note, even in the much better case you describe that would still be line coverage and I would us to improve our technique there over time.

Mike

Re: Testing Framework Site Update

[Alex Harui <ah...@adobe.com>]

On 2/28/12 6:55 AM, "Michael A. Labriola" <la...@digitalprimates.net>
wrote:

>> Correct, but as I understand them, Mustella tests are pretty specific to
>> components (e.g., display objects). Does the rest of the framework have some
>> sort of test coverage, or am I misunderstanding Mustella?
> 
> You can think of mustella tests as executing a series of actions or steps
> against an object and having the ability to wait for event to do property or
> even image comparison.  So far as I know, the only 'test coverage' numbers
> that Adobe used to define their test coverage came from running the mustella
> suite... so any other APIs were exercised by the use of mustella component
> interactions. Further, and maybe Alex can comment on this as I do not know for
> sure, I believe the test coverage information quoted by Deepa and others used
> the coverage plugin on labs they built for Flash Builder. This used the
> profiler API and was effectively fairly useless method coverage.
> 
> For example:
> 
> function doThing( val:* ):void {
> 
>  if (!val) {
> return;
> }
> 
> //5000 more lines in this method
> 
> }
> 
> Therefore if this method was called with a null, the whole method and all the
> code that did not belong in it was considered 'covered'
> 
> Mike
> 
> 
Mustella's goal was to test every line of code in the framework.  There are
very few public methods on our classes, especially the display classes, so
we would fake events to get into the entry points.  There is a mustella test
step that will call a method though.  Non-display classes like validators
and formatters and resource manager have tests, but many times those tests
are more like integration tests where the formatter or validator is wired up
to something, events are faked, and results are verified.

My understanding (needs verification) is that we used a tool that matched
the output of flash.trace.Trace to the debugline opcodes in a SWF and that
is the code coverage number.  Unless I'm missing something, that is
sufficient, and the example above would not show 100% coverage.

For sure, there aren't enough mustella tests otherwise there would be far
fewer bugs in the shipping framework, and few of these tests are unit-tests
under the strict definition, but they do a pretty good job of catching
regressions.

I am behind schedule prepping Mustella for donation.  I'm shooting for
mid-March.

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

RE: Testing Framework Site Update

["Michael A. Labriola" <la...@digitalprimates.net>]

>Correct, but as I understand them, Mustella tests are pretty specific to components (e.g., display objects). Does the rest of the framework have some sort of test coverage, or am I misunderstanding Mustella?

You can think of mustella tests as executing a series of actions or steps against an object and having the ability to wait for event to do property or even image comparison.  So far as I know, the only 'test coverage' numbers that Adobe used to define their test coverage came from running the mustella suite... so any other APIs were exercised by the use of mustella component interactions. Further, and maybe Alex can comment on this as I do not know for sure, I believe the test coverage information quoted by Deepa and others used the coverage plugin on labs they built for Flash Builder. This used the profiler API and was effectively fairly useless method coverage.

For example:

function doThing( val:* ):void {

 if (!val) {
	return;
}

//5000 more lines in this method

}

Therefore if this method was called with a null, the whole method and all the code that did not belong in it was considered 'covered'

Mike

RE: Testing Framework Site Update

["Michael A. Labriola" <la...@digitalprimates.net>]

>Hey Mike,
>
>I remember you talking about generating test stubs for all the framework classes last year on the Spoon list. Did you ever manage to finish this?
>And if so, do you think that these stubs are still something that can be of value?
>
>cheers,
>
>Roland

Not 100% finished but also not of the same value anymore. We were doing this as we could not refactor the architecture to enable testing at the time but needed some way to ensure patches weren't breaking the core framework. It was a solution for a very particular problem that I think we can now solve in a better way.

Mike

Re: Testing Framework Site Update

[Roland Zwaga <ro...@stackandheap.com>]

>
> The other concern is that we are likely to make more modification to
> existing code than to write brand new code for a while, IMO... to that end
> we are then trying to write tests against something that is very coupled
> and hence fundamentally untestable in units. When spoon first started, I
> spent months and months making surgical changes to the Flex framework to
> try to improve testability. Most of that code was approached in a very
> limited way though as we didn't have the ability to make wholesale changes
> to the framework.... given our new situation, I can be ... more aggressive
> :)
>
>
Hey Mike,

I remember you talking about generating test stubs for all the framework
classes last year on the Spoon list. Did you ever manage to finish this?
And if so, do you think that these stubs are still something that can be of
value?

cheers,

Roland

Re: Testing Framework Site Update

[Scott Delamater <sc...@gmail.com>]

For anyone else picking up this thread, my earlier question is also
answered and documented here under "Enable SDK Testability"

http://www.spoon.as/core-values/

sD

RE: Testing Framework Site Update

["Michael A. Labriola" <la...@digitalprimates.net>]

>Are those "surgical changes" in your whiteboard space? Sounds extremely valuable.

There predate Apache. The ideas may, in some cases, be valuable, however, we are now operating under different constraints to in many places I was proposing changes only because of those limitations. Example, in several cases we were working around auto-generated compiler code as, at the time, Adobe indicated they were not in any way interested in discussing compiler changes. Now, I would prefer to make those changes the right way in the compiler.

That's why I am thinking that it makes more sense to look at these differently for our new set of rules.

Mike

Re: Testing Framework Site Update

[Scott Delamater <sc...@gmail.com>]

On Tue, Feb 28, 2012 at 8:30 AM, Michael A. Labriola <
labriola@digitalprimates.net> wrote:

> The other concern is that we are likely to make more modification to
> existing code than to write brand new code for a while, IMO... to that end
> we are then trying to write tests against something that is very coupled
> and hence fundamentally untestable in units. When spoon first started, I
> spent months and months making surgical changes to the Flex framework to
> try to improve testability. Most of that code was approached in a very
> limited way though as we didn't have the ability to make wholesale changes
> to the framework.... given our new situation, I can be ... more aggressive
> :)
>

Are those "surgical changes" in your whiteboard space? Sounds extremely
valuable.

sD

RE: Testing Framework Site Update

["Michael A. Labriola" <la...@digitalprimates.net>]

>I agree, full coverage would be ideal. But I share the same concern.
>Writing unit tests at this point is mostly "oh shit does this really work"
>tests... not really the same as driving the development with them, which is why I don't see it as a high priority (for myself) to spend time on. Again, that's not to say someone else won't, just that its not how I want to prioritize my >time.

The other concern is that we are likely to make more modification to existing code than to write brand new code for a while, IMO... to that end we are then trying to write tests against something that is very coupled and hence fundamentally untestable in units. When spoon first started, I spent months and months making surgical changes to the Flex framework to try to improve testability. Most of that code was approached in a very limited way though as we didn't have the ability to make wholesale changes to the framework.... given our new situation, I can be ... more aggressive :)

Mike

Re: Testing Framework Site Update

[Omar Gonzalez <om...@gmail.com>]

>
> I'd love to see full coverage asap, but in practice I don't know how
> valuable the tests will be when writing them is decoupled from either using
> or learning the unit under test. My worry is that we'd create tests that
> gave us good coverage, but little meaning.
>
> sD
>


I agree, full coverage would be ideal. But I share the same concern.
Writing unit tests at this point is mostly "oh shit does this really work"
tests... not really the same as driving the development with them, which is
why I don't see it as a high priority (for myself) to spend time on. Again,
that's not to say someone else won't, just that its not how I want to
prioritize my time.

--
Omar Gonzalez
s9tpepper@apache.org
Apache Flex PPMC Member

Re: Testing Framework Site Update

[Scott Delamater <sc...@gmail.com>]

On Tue, Feb 28, 2012 at 7:58 AM, Omar Gonzalez <om...@gmail.com>wrote:

> No. It does not. That is why one of the biggest outstanding tasks is to
> get unit tests written for the Flex SDK. And a big reason why that hasn't
> been tackled is the highly coupled nature of the framework which will take
> quite a bit of wrangling to sort out and be able to write "real" unit
> tests.
>

I'd love to see full coverage asap, but in practice I don't know how
valuable the tests will be when writing them is decoupled from either using
or learning the unit under test. My worry is that we'd create tests that
gave us good coverage, but little meaning.

sD

Re: Testing Framework Site Update

[Omar Gonzalez <om...@gmail.com>]

On Tue, Feb 28, 2012 at 6:50 AM, Scott Delamater
<sc...@gmail.com>wrote:

> Correct, but as I understand them, Mustella tests are pretty specific to
> components (e.g., display objects). Does the rest of the framework have
> some sort of test coverage, or am I misunderstanding Mustella?
>
> sD
>
> No. It does not. That is why one of the biggest outstanding tasks is to
get unit tests written for the Flex SDK. And a big reason why that hasn't
been tackled is the highly coupled nature of the framework which will take
quite a bit of wrangling to sort out and be able to write "real" unit tests.

--
Omar Gonzalez
s9tpepper@apache.org
Apache Flex PPMC Member

Re: Testing Framework Site Update

[Scott Delamater <sc...@gmail.com>]

Correct, but as I understand them, Mustella tests are pretty specific to
components (e.g., display objects). Does the rest of the framework have
some sort of test coverage, or am I misunderstanding Mustella?

sD

On Tue, Feb 28, 2012 at 7:44 AM, Martin Heidegger <mh...@leichtgewicht.at>wrote:

> We are waiting for Mustella tests by adobe. I will start working on
> FlexUnit tests next time I see fit.
>
> yours
> Martin.
>
>
> On 28/02/2012 23:43, Scott Delamater wrote:
>
>> On Fri, Feb 17, 2012 at 12:25 PM, Alex Harui<ah...@adobe.com>  wrote:
>>
>>  Yes the Marshall Plan is not about units, but I assume there is some
>>> vision
>>> for how to do "integration testing" or "functional testing".  I thought
>>> FlexUnit had the ability to do more than just "unit tests". Or do you
>>> think
>>> we can ship a valid release just by unit tests?
>>>
>>
>> I assume I'm missing something, but I don't see *any* unit tests in the
>> repo. Are we still waiting on those?
>>
>> sD
>>
>>
>

Re: Testing Framework Site Update

[Martin Heidegger <mh...@leichtgewicht.at>]

We are waiting for Mustella tests by adobe. I will start working on 
FlexUnit tests next time I see fit.

yours
Martin.

On 28/02/2012 23:43, Scott Delamater wrote:
> On Fri, Feb 17, 2012 at 12:25 PM, Alex Harui<ah...@adobe.com>  wrote:
>
>> Yes the Marshall Plan is not about units, but I assume there is some vision
>> for how to do "integration testing" or "functional testing".  I thought
>> FlexUnit had the ability to do more than just "unit tests". Or do you think
>> we can ship a valid release just by unit tests?
>
> I assume I'm missing something, but I don't see *any* unit tests in the
> repo. Are we still waiting on those?
>
> sD
>

Re: Testing Framework Site Update

[Scott Delamater <sc...@gmail.com>]

On Fri, Feb 17, 2012 at 12:25 PM, Alex Harui <ah...@adobe.com> wrote:

>
> Yes the Marshall Plan is not about units, but I assume there is some vision
> for how to do "integration testing" or "functional testing".  I thought
> FlexUnit had the ability to do more than just "unit tests". Or do you think
> we can ship a valid release just by unit tests?


I assume I'm missing something, but I don't see *any* unit tests in the
repo. Are we still waiting on those?

sD

RE: Testing Framework Site Update

["Michael A. Labriola" <la...@digitalprimates.net>]

>Yes the Marshall Plan is not about units, but I assume there is some vision for how to do "integration testing" or "functional testing".  I thought FlexUnit had the ability to do more than just "unit tests". Or do you think we can ship >a valid release just by unit tests?

Alex,

I don't think we can ship a valid release by just unit testing. I think unit testing is one of many types of tests we need for a valid release. That's why I still think mustella is very important. I am sure we can figure out a way to work on something that can do what we need. FlexUnit runs tests in isolation, prototype version actually runs all tests in their own domain, but right now it doesn't have the idea of bridging a test run across multiple invocations of the Flash Player. All tests are done within a single invocation. So, just something to be brainstormed as well.

Cheers,
Mike
 

Re: Testing Framework Site Update

[Alex Harui <ah...@adobe.com>]

Mustella has a bitmap compare feature.  We get anti-aliasing differences on
different platforms.


On 2/17/12 11:35 AM, "David Francis Buhler" <da...@gmail.com> wrote:

> I'm curious to know the differences Mustella measures when comparing
> test-results for the Mac OS vs. Windows OS. How much do the SDK classes
> vary in their behavior on different Operating Systems?
> 
> On Fri, Feb 17, 2012 at 2:25 PM, Alex Harui <ah...@adobe.com> wrote:
> 
>> 
>> 
>> 
>> On 2/17/12 9:59 AM, "Michael A. Labriola" <la...@digitalprimates.net>
>> wrote:
>> 
>> 
>>> 
>>> FlexUnit won't be able to support that as nothing about the Marshall
>> Plan is a
>>> unit. It is running a whole additional application in a whole additional
>>> domain. It is sort of the antithesis of what FlexUnit is trying to
>> accomplish.
>>> 
>> Yes the Marshall Plan is not about units, but I assume there is some vision
>> for how to do "integration testing" or "functional testing".  I thought
>> FlexUnit had the ability to do more than just "unit tests". Or do you think
>> we can ship a valid release just by unit tests?
>> 
>> --
>> Alex Harui
>> Flex SDK Team
>> Adobe Systems, Inc.
>> http://blogs.adobe.com/aharui
>> 
>> 

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

Re: Testing Framework Site Update

[David Francis Buhler <da...@gmail.com>]

I'm curious to know the differences Mustella measures when comparing
test-results for the Mac OS vs. Windows OS. How much do the SDK classes
vary in their behavior on different Operating Systems?

On Fri, Feb 17, 2012 at 2:25 PM, Alex Harui <ah...@adobe.com> wrote:

>
>
>
> On 2/17/12 9:59 AM, "Michael A. Labriola" <la...@digitalprimates.net>
> wrote:
>
>
> >
> > FlexUnit won't be able to support that as nothing about the Marshall
> Plan is a
> > unit. It is running a whole additional application in a whole additional
> > domain. It is sort of the antithesis of what FlexUnit is trying to
> accomplish.
> >
> Yes the Marshall Plan is not about units, but I assume there is some vision
> for how to do "integration testing" or "functional testing".  I thought
> FlexUnit had the ability to do more than just "unit tests". Or do you think
> we can ship a valid release just by unit tests?
>
> --
> Alex Harui
> Flex SDK Team
> Adobe Systems, Inc.
> http://blogs.adobe.com/aharui
>
>

Re: Testing Framework Site Update

[Alex Harui <ah...@adobe.com>]

On 2/17/12 9:59 AM, "Michael A. Labriola" <la...@digitalprimates.net>
wrote:


> 
> FlexUnit won't be able to support that as nothing about the Marshall Plan is a
> unit. It is running a whole additional application in a whole additional
> domain. It is sort of the antithesis of what FlexUnit is trying to accomplish.
> 
Yes the Marshall Plan is not about units, but I assume there is some vision
for how to do "integration testing" or "functional testing".  I thought
FlexUnit had the ability to do more than just "unit tests". Or do you think
we can ship a valid release just by unit tests?

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

RE: Testing Framework Site Update

["Michael A. Labriola" <la...@digitalprimates.net>]

>At one point, there was a discussion with the FlexUnit folks of taking some aspects of Mustella and enhancing FlexUnit to support things like Marshall Plan.  It might be worth looking into that again.

FlexUnit won't be able to support that as nothing about the Marshall Plan is a unit. It is running a whole additional application in a whole additional domain. It is sort of the antithesis of what FlexUnit is trying to accomplish.

Mike

Re: Testing Framework Site Update

[Omar Gonzalez <om...@gmail.com>]

On Thursday, February 16, 2012, Alex Harui wrote:

>
>
>
> On 2/16/12 6:02 PM, "Omar Gonzalez" <omarg.developer@gmail.com<javascript:;>>
> wrote:
>
> > We use [FlexUnit][4] for unit testing and [Mockolate][5] for mocks and
> test
> > spies.
> Are we actually using or planning to use?
>
> I have not researched any of these test frameworks, but I didn't think any
> of them handled Marshall Plan configurations.
>
> At one point, there was a discussion with the FlexUnit folks of taking some
> aspects of Mustella and enhancing FlexUnit to support things like Marshall
> Plan.  It might be worth looking into that again.
>
> --
> Alex Harui
> Flex SDK Team
> Adobe Systems, Inc.
> http://blogs.adobe.com/aharui



Well, I also included text that said we'd have more info on Mustella itself
for functional testing. Since it hasn't been donated, and I don't know the
specifics of what it offers, I wrote on the page that we'd post more info
about Mustella as it becomes available. If it does what I think it does
we'd likely be using all three. But I couldn't write that on the site
without specifics.

-omar

Re: Testing Framework Site Update

[Omar Gonzalez <om...@gmail.com>]

On Thursday, February 16, 2012, Alex Harui wrote:

>
>
>
> On 2/16/12 6:02 PM, "Omar Gonzalez" <omarg.developer@gmail.com<javascript:;>>
> wrote:
>
> > We use [FlexUnit][4] for unit testing and [Mockolate][5] for mocks and
> test
> > spies.
> Are we actually using or planning to use?
>
> I have not researched any of these test frameworks, but I didn't think any
> of them handled Marshall Plan configurations.
>
> At one point, there was a discussion with the FlexUnit folks of taking some
> aspects of Mustella and enhancing FlexUnit to support things like Marshall
> Plan.  It might be worth looking into that again.
>
> --
> Alex Harui
> Flex SDK Team
> Adobe Systems, Inc.
> http://blogs.adobe.com/aharui



Well, I also included text that said we'd have more info on Mustella itself
for functional testing. Since it hasn't been donated, and I don't know the
specifics of what it offers, I wrote on the page that we'd post more info
about Mustella as it becomes available. If it does what I think it does
we'd likely be using all three. But I couldn't write that on the site
without specifics.

-omar

RE: Testing Framework Site Update

["Michael A. Labriola" <la...@digitalprimates.net>]

>@Michael Labriola Is there any progress? Open Questions?

Heard back from the mentor yesterday. There will be many open questions, still working through the requirements side of the IP clearance....

Mike

Re: Testing Framework Site Update

[Martin Heidegger <mh...@leichtgewicht.at>]

On 17/02/2012 15:18, Alex Harui wrote:
> Are we actually using or planning to use? I have not researched any of 
> these test frameworks, but I didn't think any of them handled Marshall 
> Plan configurations. At one point, there was a discussion with the 
> FlexUnit folks of taking some aspects of Mustella and enhancing 
> FlexUnit to support things like Marshall Plan. It might be worth 
> looking into that again. 
There was a long discussion about it, ending with a 72hr deadline to a 
lazy consensus: If no one answers its accepted given.  As FlexUnit seems 
to be in stage of contribution to Flex: We will be "the FlexUnit folks".

@Michael Labriola Is there any progress? Open Questions?

yours
Martin.

Re: Testing Framework Site Update

[Alex Harui <ah...@adobe.com>]

On 2/16/12 6:02 PM, "Omar Gonzalez" <om...@gmail.com> wrote:

> We use [FlexUnit][4] for unit testing and [Mockolate][5] for mocks and test
> spies.
Are we actually using or planning to use?

I have not researched any of these test frameworks, but I didn't think any
of them handled Marshall Plan configurations.

At one point, there was a discussion with the FlexUnit folks of taking some
aspects of Mustella and enhancing FlexUnit to support things like Marshall
Plan.  It might be worth looking into that again.

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui