Unit tests

[Om <bi...@gmail.com>]

Is there an effort to start writing unit tests for the Flex SDK?  I know
that the Mustella suite is coming, but I have heard calls for a real unit
test suite.  I would love to contribute by writing as many unit tests as
possible for the existing SDK code.  I realize that some portions are
un-'unit-testable', but we can sure write tests for the ones that we can,
right?

I am not a committer, so am wondering what would be the best way to go
about starting this process?  Anyone else interested?  Michael ;-) ?

Thanks,
Om

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>Yeah maybe.  Are you committing to doing documentation and support?  Because if I do get the steps out there sooner, I don't want to be distracted by answering questions.

Yes, that was my plan. I would also work on the integration within the FlexUnit build and test process so we could have a unified test run from the build servers eventually.

>Right now, donating the mustella steps would be blocked by the same legal issue blocking the MXMLC compiler.  We'll see how far I am with the rest of mustella when we get the compiler donated and decide on a plan.

Understood.

Re: Unit tests

[Alex Harui <ah...@adobe.com>]

On 3/13/12 7:10 PM, "Michael A. Labriola" <la...@digitalprimates.net>
wrote:

> Just thinking it would give people a way to start writing tests in this
> supported way so that we are getting coverage on new code. I might put this on
> my plate.
Yeah maybe.  Are you committing to doing documentation and support?  Because
if I do get the steps out there sooner, I don't want to be distracted by
answering questions.

Right now, donating the mustella steps would be blocked by the same legal
issue blocking the MXMLC compiler.  We'll see how far I am with the rest of
mustella when we get the compiler donated and decide on a plan.

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>Many folks want to see all the mustella tests in before making changes to the SDK which is another reason why I've been trying to do it in all one shot.

Just thinking it would give people a way to start writing tests in this supported way so that we are getting coverage on new code. I might put this on my plate.

Re: Unit tests

[Alex Harui <ah...@adobe.com>]

On 3/13/12 11:52 AM, "Michael A. Labriola" <la...@digitalprimates.net>
wrote:

>> The real #2 has a bunch of optional parameters and doesn't rely on
>> flashlog.txt
> 
> Just wondering if the fake #2 still gets us started for now.
Well, there really isn't anything in a fake #2.  It is just a matter of
compiling an MXML file with tags from #1.  If you look in the checkintests
folder you'll see a -config.xml that adds the script MXML files to the main
MXML file.

So, if you want to manually build the SWFs, run them and grab flashlog.txt
and sniff it for results you could get started with just #1.

There will be more work if you want to use a bitmap compare in a test.

Many folks want to see all the mustella tests in before making changes to
the SDK which is another reason why I've been trying to do it in all one
shot.

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>The real #2 has a bunch of optional parameters and doesn't rely on flashlog.txt

Just wondering if the fake #2 still gets us started for now.

Re: Unit tests

[Alex Harui <ah...@adobe.com>]

On 3/13/12 11:09 AM, "Michael A. Labriola" <la...@digitalprimates.net>
wrote:

> 
>> 1) A SWC of test steps that are tags in MXML used to define the script.
>> 2) A Java engine that runs tests based on command line input
> 
> Isn't some version of #1 and #2 in the check in tests?
#2 is not in the checkintests.  The build.xml in the checkintests folder
simply compiles the scripts with the swc and launches it, then parses the
flashlog.txt looking for errors.

The real #2 has a bunch of optional parameters and doesn't rely on
flashlog.txt


-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>1) A SWC of test steps that are tags in MXML used to define the script.
>2) A Java engine that runs tests based on command line input

Isn't some version of #1 and #2 in the check in tests?

Re: Unit tests

[Omar Gonzalez <s9...@apache.org>]

On Tue, Mar 13, 2012 at 9:54 AM, JP Bader <jp...@zavteq.com> wrote:

> Every time we want a little progress, Adobe makes sure their "legal"
> has some more wrangling to do, so we fall a little further
> behind...Does anyone else sense that maybe Adobe is intentionally
> sabotaging any of our progress?  That's just the conspiracy-theorist
> in me...
>
>
I don't think so, and I don't think starting up conspiracy theory threads
is conducive to the people that are working toward getting us past that,
namely Alex and Carol. I am just as frustrated as the next person, and I'm
sure Alex and Carol are even more so since they're the ones doing the work.
But as it's been brought up already this is not uncommon for projects with
extremely large code bases and history to migrate. All legal departments
are slow, they get paid by the hour.. :P

The best we can do is to have patience and work on things that are possible
to work on, such as Tink and Justin have been doing. I am waiting for JIRA
issues to get imported to work on existing SDK, in the meanwhile I have
some experiments going on that I'm trying to get to a place where I feel
comfortable sharing with everyone so they can rip it apart and start some
iterative cycles on.

Let's try to stay positive! (I know I'm definitely guilty of not being
positive sometimes, but I'm making an effort)

-- 
Omar Gonzalez
s9tpepper@apache.org

Re: Unit tests

[João Fernandes <jo...@gmail.com>]

On 13 March 2012 16:54, JP Bader <jp...@zavteq.com>
>
> behind...Does anyone else sense that maybe Adobe is intentionally
> sabotaging any of our progress?  That's just the conspiracy-theorist
> in me...
>
>
Clearly conspiracy-theory. They are trying hard to get everything donated
and if you look in the archives, you'll see that they are trying their best
to get everything donated as soon as possible. Of course Legal clearance is
the worst step but anyone used to deal with that kind of stuff knows that
it's a nightmare, so yes, in my opinion they aren't trying to sabotage
anything.


-- 

João Fernandes

Re: Unit tests

[JP Bader <jp...@zavteq.com>]

I concede, it's my own paranoia...

I am stoked, love Tink's and Justin's contributions, and look forward
to committing shortly.

On Tue, Mar 13, 2012 at 12:24 PM, Michael A. Labriola
<la...@digitalprimates.net> wrote:
>>Every time we want a little progress, Adobe makes sure their "legal"
>>has some more wrangling to do, so we fall a little further behind...Does anyone else sense that maybe Adobe is intentionally sabotaging any of our progress?  That's just the conspiracy-theorist in me...
>
>
> Not with you there. I think this is legit, its just unfortunate
>



-- 
JP Bader
Principal
Zavteq, Inc.
@lordB8r | jp@zavteq.com
608.692.2468

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>Every time we want a little progress, Adobe makes sure their "legal"
>has some more wrangling to do, so we fall a little further behind...Does anyone else sense that maybe Adobe is intentionally sabotaging any of our progress?  That's just the conspiracy-theorist in me...


Not with you there. I think this is legit, its just unfortunate

Re: Unit tests

[Carol Frampton <cf...@adobe.com>]

>
>
>
>If you take this road you could also argue that Apache is deliberately
>delaying because we we're waiting long for the Jira import. And that is
>of course ridiculous.

Apache infrastructure ended up filing a support issue at Atlassian so they
are waiting for support from them.  Our bug is
https://issues.apache.org/jira/browse/INFRA-4380 and there is a link to
the Atlassian bug but it doesn't seem to be viewable without an account

Carol

Re: Unit tests

[Arnoud Bos <ar...@artim-interactive.nl>]

On Mar 13, 2012, at 5:54 PM, JP Bader wrote:

> Every time we want a little progress, Adobe makes sure their "legal"
> has some more wrangling to do, so we fall a little further
> behind...Does anyone else sense that maybe Adobe is intentionally
> sabotaging any of our progress?  That's just the conspiracy-theorist
> in me...
> 

It crossed my mind once too, but thinking about it, it doesn't make sense.

I think it's better for Adobe if we succeed. Cause more people will target
their runtimes and also we can provide a way to keep their Flex customers
happy.  On the other hand... They make most money with tools
and for Flex they are kind of pulling the plug with FlashBuilder for Flex.

So i guess it's just not as high a priority for them as it is for us. It's a big company
and their legal dept. probably has more on their plate than Apache Flex. 
But that they are deliberately delaying might be a bit too far fetched. 

If you take this road you could also argue that Apache is deliberately
delaying because we we're waiting long for the Jira import. And that is
of course ridiculous.

Arnoud

Re: Unit tests

[JP Bader <jp...@zavteq.com>]

Every time we want a little progress, Adobe makes sure their "legal"
has some more wrangling to do, so we fall a little further
behind...Does anyone else sense that maybe Adobe is intentionally
sabotaging any of our progress?  That's just the conspiracy-theorist
in me...

On Tue, Mar 13, 2012 at 11:40 AM, Michael A. Labriola
<la...@digitalprimates.net> wrote:
>>Adobe probably won't go after folks who have legal copies of mustella.swc and guess how it works and write and
> run tests on their computers, but I don't mustella.swc can get checked into Apache without Adobe's approval.
>
> Okay, so if I post a decompiled version of the SWC on github with commentary? That was a little sarcastic, just not sure how to proceed.
>
> Mike
>



-- 
JP Bader
Principal
Zavteq, Inc.
@lordB8r | jp@zavteq.com
608.692.2468

Re: Unit tests

[Martin Heidegger <mh...@leichtgewicht.at>]

On 14/03/2012 03:07, Michael A. Labriola wrote:
> The AIR stuff has some of its own complexity too. Ideally we would like to test things in Flash Player and in AIR, not just one or the othe
My thinking is following: Right now X images are compressed into & 
decompressed from PNG. I assume the Flex tests have more than 10 
pictures and I can imagine
that this just takes a while by itself. I guess without this overhead it 
would be slightly faster to test things, no?

yours
Martin.

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>Yeah, maybe someday we can rewrite the engine to use AIR instead of Java, but mustella was around long before ANEs.

The AIR stuff has some of its own complexity too. Ideally we would like to test things in Flash Player and in AIR, not just one or the other. 

Re: Unit tests

[Alex Harui <ah...@adobe.com>]

On 3/13/12 10:52 AM, "Martin Heidegger" <mh...@leichtgewicht.at> wrote:

> Just out of curiosity: It would be no problem to run a test in adl an
> capture the output there using a native extension, would it?
> I am not sure how the tests are built but making a screenshot is of a
> application is not "high science" and it could be easier to setup
> a browser/server/etc stuff.
Yeah, maybe someday we can rewrite the engine to use AIR instead of Java,
but mustella was around long before ANEs.

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

Re: Unit tests

[Martin Heidegger <mh...@leichtgewicht.at>]

On 14/03/2012 02:46, Alex Harui wrote:
>
> Well, sarcastic or not, it is a good question.
>
> Here's my latest thinking:  There are several pieces to mustella.
>
> 1) A SWC of test steps that are tags in MXML used to define the script.
> 2) A Java engine that runs tests based on command line input
> 3) An http server that captures output from the tests
> 4) A ton of test scripts
>
> The checkintests in the full Adobe SDK only include #1 and a few tests.  The
> don't use any bitmap compares because you need #3 to save away the baseline
> you are going to compare against.
>
> My current thinking is that I need to legally clear and donate all four in
> one big chunk so that:
> A) I only have to bug legal once
> B) I only have to bug the execs who have to sign off on this stuff once
> C) It is clear what tests have been written so folks don't go duplicating
> work
> D) It is easier to run a subset of tests via the command line
>
> I'm open to arguments that say I should try to get #1 done sooner at the
> expense of delaying #2, #3 and #4.

Just out of curiosity: It would be no problem to run a test in adl an 
capture the output there using a native extension, would it?
I am not sure how the tests are built but making a screenshot is of a 
application is not "high science" and it could be easier to setup
a browser/server/etc stuff.

yours
Martin.

Re: Unit tests

[Alex Harui <ah...@adobe.com>]

On 3/13/12 9:40 AM, "Michael A. Labriola" <la...@digitalprimates.net>
wrote:

>> Adobe probably won't go after folks who have legal copies of mustella.swc and
>> guess how it works and write and
> run tests on their computers, but I don't mustella.swc can get checked into
> Apache without Adobe's approval.
> 
> Okay, so if I post a decompiled version of the SWC on github with commentary?
> That was a little sarcastic, just not sure how to proceed.
> 
Well, sarcastic or not, it is a good question.

Here's my latest thinking:  There are several pieces to mustella.

1) A SWC of test steps that are tags in MXML used to define the script.
2) A Java engine that runs tests based on command line input
3) An http server that captures output from the tests
4) A ton of test scripts

The checkintests in the full Adobe SDK only include #1 and a few tests.  The
don't use any bitmap compares because you need #3 to save away the baseline
you are going to compare against.

My current thinking is that I need to legally clear and donate all four in
one big chunk so that:
A) I only have to bug legal once
B) I only have to bug the execs who have to sign off on this stuff once
C) It is clear what tests have been written so folks don't go duplicating
work
D) It is easier to run a subset of tests via the command line

I'm open to arguments that say I should try to get #1 done sooner at the
expense of delaying #2, #3 and #4.

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>Adobe probably won't go after folks who have legal copies of mustella.swc and guess how it works and write and 
run tests on their computers, but I don't mustella.swc can get checked into Apache without Adobe's approval.

Okay, so if I post a decompiled version of the SWC on github with commentary? That was a little sarcastic, just not sure how to proceed.

Mike

Re: Unit tests

[Alex Harui <ah...@adobe.com>]

On 3/13/12 6:35 AM, "Michael A. Labriola" <la...@digitalprimates.net>
wrote:
> 
> Alex,
> 
> Let's just say I happen to have an older version of the source... If I was
> willing to support this on my own and deal with questions, do you have any
> problem with people writing tests against mustella. We cant write unit tests
> until we refactor, but we cant write mustella tests because we don't have the
> source... its an evil place to be. I want to get us out.
I want to get us out too, but I'm not sure how.

If your version of the source was provided under NDA, then I don't think you
can discuss its contents publicly.

Adobe probably won't go after folks who have legal copies of mustella.swc
and guess how it works and write and run tests on their computers, but I
don't mustella.swc can get checked into Apache without Adobe's approval.

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>I think it is better to wait.  Folks could probably play around on their own with the Adobe mustella.swc, but I don't want to take the time to deal with questions and doc and bugs right now.

Alex,

Let's just say I happen to have an older version of the source... If I was willing to support this on my own and deal with questions, do you have any problem with people writing tests against mustella. We cant write unit tests until we refactor, but we cant write mustella tests because we don't have the source... its an evil place to be. I want to get us out.

Mike

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>Do we have any info on how to write and run Mustella tests? Do we have access to the framework smoke tests?

I don't think they are checked in yet, but if you were to grab the code from Adobe's SVN, the framework smoke tests run as part of the check in target.

>I don't even know what one looks like or what the differences are from FlexUnit other than in a general way.

They are actually really nice for this type of test, they are all written in MXML and represent actions and wait states. They are similar in some ways to the Sequence stuff in Fluint or FlexUnit 4 but each one can run in a unique instance of flash player so you can better test things like styles, etc.

Mike

Re: Unit tests

[Justin Mclean <ju...@classsoftware.com>]

Hi,

>  the framework smoke tests worked on mustella. I mention this because the tests you list above are much better as mustella tests than as flexunit tests.
Do we have any info on how to write and run Mustella tests? Do we have access to the framework smoke tests?

I don't even know what one looks like or what the differences are from FlexUnit other than in a general way.

Justin

Re: Unit tests

[Alex Harui <ah...@adobe.com>]

On 3/12/12 6:55 PM, "Michael A. Labriola" <la...@digitalprimates.net>
wrote:

> Alex, am I wrong? Can we use mustella even without the Adobe test cases?
We have not checked any mustella into Apache.  And I'm not seeing that there
is a mustella.swc in the Adobe Open Source SDK release.  I think it is in
the not-fully-open release, but there isn't any official doc on it.

I think it is better to wait.  Folks could probably play around on their own
with the Adobe mustella.swc, but I don't want to take the time to deal with
questions and doc and bugs right now.

-- 
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>I've been able to do some tests on the datagrid by fudging the focus manager a little. I've used it to test my own item renders/editors and modifications to the datagrids in the past. It may be useful for testing the >grid for submissions/patches. I have some code lying about that I can put up on the whiteboard if anyone is interested. 


I am not saying these aren't useful, I am just saying they aren't unit tests. However, semantics aside, I have a question... 

I understand that we can't have all of Adobe's mustella tests due to IP but, so far as I know, there is no reason we can't start writing our own mustella tests today... the framework smoke tests worked on mustella. I mention this because the tests you list above are much better as mustella tests than as flexunit tests.

Alex, am I wrong? Can we use mustella even without the Adobe test cases?

Mike

Re: Unit tests

[Justin Mclean <ju...@classsoftware.com>]

Hi,

> I am not trying to discourage anyone, just manage expectations. I promise pain and disappointment awaits if you decide to 'unit test' DataGrid.

I've been able to do some tests on the datagrid by fudging the focus manager a little. I've used it to test my own item renders/editors and modifications to the datagrids in the past. It may be useful for testing the grid for submissions/patches. I have some code lying about that I can put up on the whiteboard if anyone is interested. 

The unit tests be not be perfect done this way but they may be good enough and any tests are better than none right?

Justin

RE: Unit tests

["Michael A. Labriola" <la...@digitalprimates.net>]

>I am not a committer, so am wondering what would be the best way to go about starting this process?  Anyone else interested?  Michael ;-) ?

There are some places you can write unit tests in the SDK... however, and this will remain the big problem for quite some time. Things are way too coupled to write real unit tests for anything remotely resembling a component and even a fair number of the supporting classes. There was some discussion in the beginning of the Apache Flex list about packages that could be unit tested. I think it would be a great and useful effort if someone wanted to start there. I am not trying to discourage anyone, just manage expectations. I promise pain and disappointment awaits if you decide to 'unit test' DataGrid.

Mike

Re: Unit tests

[Martin Heidegger <mh...@leichtgewicht.at>]

On 13/03/2012 08:16, Om wrote:
> Is there an effort to start writing unit tests for the Flex SDK?  I know
> that the Mustella suite is coming, but I have heard calls for a real unit
> test suite.  I would love to contribute by writing as many unit tests as
> possible for the existing SDK code.  I realize that some portions are
> un-'unit-testable', but we can sure write tests for the ones that we can,
> right?
>
> I am not a committer, so am wondering what would be the best way to go
> about starting this process?  Anyone else interested?  Michael ;-) ?
>
> Thanks,
> Om

I think i mentioned before, next time I sit on Flex code it will be unit 
tests :)
my way to start the process: Write :)

yours
Martin.