You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flex.apache.org by Erik de Bruin <er...@ixsoftware.nl> on 2013/04/12 16:02:38 UTC
Meet 'marmotinni', an ASJS testing framework
Hi,
I've just committed a new testing framework to the ASJS repo. This
testing framework, which I've called 'marmotinni', following the
'mustella' convention (Google it ;-), consists of FalconJX, asjs
projects, Java test classes (using the Selenium browser testing
classes) and an ant script to tie them all together.
Right now there's only one test. It consists of a FlexJS project that
displays a single TextButton 100px from the left side of the browser
window. The ant script builds a JS release version using FalconJX. The
'TextButton' Java class uses the Selenium framework to display the
FlexJS release build in a Firefox window, see if the button is there
and in the correct position. The result of the test is passed back to
ant and all artefacts are cleaned up.
Please take a look and let me know what you think.
EdB
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
RE: Meet 'marmotinni', an ASJS testing framework
Posted by Tigran Najaryan <ti...@gmail.com>.
> From: Erik de Bruin [mailto:erik@ixsoftware.nl]
>
> Hi,
>
> I've just committed a new testing framework to the ASJS repo. This testing
> framework, which I've called 'marmotinni', following the 'mustella'
> convention (Google it ;-), consists of FalconJX, asjs projects, Java test
classes
> (using the Selenium browser testing
> classes) and an ant script to tie them all together.
>
> Right now there's only one test. It consists of a FlexJS project that
displays a
> single TextButton 100px from the left side of the browser window. The ant
> script builds a JS release version using FalconJX. The 'TextButton' Java
class
> uses the Selenium framework to display the FlexJS release build in a
Firefox
> window, see if the button is there and in the correct position. The result
of
> the test is passed back to ant and all artefacts are cleaned up.
>
> Please take a look and let me know what you think.
I tried to run 'marmotinni' to see the test.
I run the Ant build and it passed all the preparation compilation steps then
on the "test" step it opened a Firefox with blank URL, then nothing
happened, the browser window is stuck open, the Ant output shows "[echo]
Testing...".
It looks like for some reason the Selenium driver did not load the URL.
Tigran.
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
This is not for unit testing. This is for functional testing FlexJS,
like mustella is for functional testing the Flex SDK. Seemed like a
worthwhile thing to have...
EdB
On Fri, Apr 12, 2013 at 6:17 PM, christofer.dutz@c-ware.de
<ch...@c-ware.de> wrote:
> I would guess that when compiling to JS FlexUnit won't work as FlexUnit requires the Unit Tests to run in a FlashPlayer instance ... Am I correct? So without a unit-testing framework to support JS Flex, I guess we couldn't actually do any Unit Tests with it.
>
> Guess this problem is similar to the debugging problem as we currently wouldn't have any decent Debugging support to debug JS Flex applications from inside the IDE.
>
> Chris
> ________________________________________
> Von: Alex Harui [aharui@adobe.com]
> Gesendet: Freitag, 12. April 2013 18:14
> An: dev@flex.apache.org
> Betreff: Re: Meet 'marmotinni', an ASJS testing framework
>
> I haven't really looked yet, but why aren't we going to use FlexUnit?
>
>
> On 4/12/13 7:02 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
>
>> Hi,
>>
>> I've just committed a new testing framework to the ASJS repo. This
>> testing framework, which I've called 'marmotinni', following the
>> 'mustella' convention (Google it ;-), consists of FalconJX, asjs
>> projects, Java test classes (using the Selenium browser testing
>> classes) and an ant script to tie them all together.
>>
>> Right now there's only one test. It consists of a FlexJS project that
>> displays a single TextButton 100px from the left side of the browser
>> window. The ant script builds a JS release version using FalconJX. The
>> 'TextButton' Java class uses the Selenium framework to display the
>> FlexJS release build in a Firefox window, see if the button is there
>> and in the correct position. The result of the test is passed back to
>> ant and all artefacts are cleaned up.
>>
>> Please take a look and let me know what you think.
>>
>> EdB
>>
>>
>>
>> --
>> Ix Multimedia Software
>>
>> Jan Luykenstraat 27
>> 3521 VB Utrecht
>>
>> T. 06-51952295
>> I. www.ixsoftware.nl
>
> --
> Alex Harui
> Flex SDK Team
> Adobe Systems, Inc.
> http://blogs.adobe.com/aharui
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
AW: Meet 'marmotinni', an ASJS testing framework
Posted by "christofer.dutz@c-ware.de" <ch...@c-ware.de>.
I would guess that when compiling to JS FlexUnit won't work as FlexUnit requires the Unit Tests to run in a FlashPlayer instance ... Am I correct? So without a unit-testing framework to support JS Flex, I guess we couldn't actually do any Unit Tests with it.
Guess this problem is similar to the debugging problem as we currently wouldn't have any decent Debugging support to debug JS Flex applications from inside the IDE.
Chris
________________________________________
Von: Alex Harui [aharui@adobe.com]
Gesendet: Freitag, 12. April 2013 18:14
An: dev@flex.apache.org
Betreff: Re: Meet 'marmotinni', an ASJS testing framework
I haven't really looked yet, but why aren't we going to use FlexUnit?
On 4/12/13 7:02 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
> Hi,
>
> I've just committed a new testing framework to the ASJS repo. This
> testing framework, which I've called 'marmotinni', following the
> 'mustella' convention (Google it ;-), consists of FalconJX, asjs
> projects, Java test classes (using the Selenium browser testing
> classes) and an ant script to tie them all together.
>
> Right now there's only one test. It consists of a FlexJS project that
> displays a single TextButton 100px from the left side of the browser
> window. The ant script builds a JS release version using FalconJX. The
> 'TextButton' Java class uses the Selenium framework to display the
> FlexJS release build in a Firefox window, see if the button is there
> and in the correct position. The result of the test is passed back to
> ant and all artefacts are cleaned up.
>
> Please take a look and let me know what you think.
>
> EdB
>
>
>
> --
> Ix Multimedia Software
>
> Jan Luykenstraat 27
> 3521 VB Utrecht
>
> T. 06-51952295
> I. www.ixsoftware.nl
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
Ok, just so we're on the same topic: I'm talking about functional
tests of the output of the compiler. This means that I want to be able
to test if a button with an event that changes the text on a label
that we've created using the FlexJS AS framework does, after being put
through the FalconJX compiler, actually works using the FlexJS JS
framework in IE, Firefox and Chrome. This means that the test
framework needs to be able to take a MXML/AS application, compile it,
launch the output in a browser, fake a click on the button and check
the text of the label to see if it has the correct value.
EdB
On Fri, Apr 12, 2013 at 8:31 PM, Alex Harui <ah...@adobe.com> wrote:
>
>
>
> On 4/12/13 11:12 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
>
>>> I don't know much about testing frameworks other than mustella, so I'm
>>> mostly asking dumb questions. I don't have a fully-formed plan on how to
>>> test all of this stuff, so I'm not saying you made the wrong decision. I'm
>>> just curious. How do I test the AS Side? Does this mean I have two
>>> versions of each test?
>>
>> With this solution, you don't test the AS side (yet). Unless we write
>> a meta-layer that is interpreted by both an AS functional testing
>> framework (mustella?) and a JS functional testing framework
>> (marmotinni?), there is no framework Google knows about that does
>> functional testing on both AS and JS based on the same test data/file.
> I would like to explore what that layer looks like before we author a bunch
> of tests that have to be duplicated to test the AS side. I still think
> FlexUnit can do functional testing just like Junit can, but I could be
> wrong.
>>
>>> I was just looking at the compiler.jx.tests trying to figure out where to
>>> add more tests. These seem like feature tests to me, do you agree? Yet we
>>> are using Junit to run them. I haven't used FlexUnit but I assume it can do
>>> unit and functional testing just like Junit. And we own the code so we can
>>> make it do that if needed.
>>
>> Feature tests? If by that you mean functional tests, then no, I don't
>> agree. The tests in compiler.jx.tests are unit tests that test if the
>> output string from FalconJX matches the expected value, given a
>> certain input. It in no way tests whether the input or the output
>> actually DOES anything. I doesn't check if given the code for a click
>> on a button, the code correctly fires an event that causes the text of
>> a specified label to change to the correct value. That is functional
>> testing, IMHO.
> Yeah, I meant functional testing. IMHO, these are functional tests. They
> test that the compiler is functioning as a whole. It isn't testing the
> individual units like a set of tests for the JSEmitter.java class. And a
> separate suite of functional tests for ASJS would probably live in the asjs
> repo, not the falcon repo. But I could certainly be wrong here.
>>
>>> I do think I just want to write the "source" for a test once and then have
>>> it run "everywhere".
>>
>> That would be ideal, and maybe we can use the format that mustella
>> uses (MXML if I'm correct?) and make marmotinni use the same format,
>> making if possible to use the same functional tests for both the AS
>> (Flash Player) side and the JS (browser) side.
> IMO, MXML or AS. We're in the business of translating either to JS aren't
> we?
>>
>>> As I've been going through the compiler.jx.tests, it makes me want to
>>> refactor those tests as well so they can be used to test Falcon's AS
>>> handling too.
>>
>> The tests in 'org.apache.flex.compiler.internal.codegen.as' test
>> exactly that. They test AS to AS compilation of most language features
>> (some work needs to be done writing tests for the missing feature).
>> All other ASJS test 'suites' (amd, goog, flexjs) subclass these tests
>> and only override them where the expected output differs from that of
>> the parent. It was set up this way to ensure that starting with
>> testing all possible AS output, all the language features would also
>> be tested in the subclasses, i.e. the various JS output formats.
> Yeah, but it seems like I have to override by copying the entire test
> including its source instead of having a set of source inputs and an
> overridable set of expected outputs. That's what I was thinking of
> refactoring.
>>
>>> At one point several years ago, Mike L and I discussed authoring FlexUnit
>>> tests in MXML. I still like the idea as it more tightly controls the way
>>> tests are authored.
>>
>> As I mentioned above, we might look into mustella's MXML formatted
>> tests and see if we can leverage that in marmotinni. I don't think
>> FlexUnit (being a unit testing framework) will be able to run
>> functional tests in both the Flash Player and the various browsers...
> Well, we own the code for FlexUnit, so at the point it takes the compiler
> output and launches the player and tears it down (I am just assuming it does
> that) maybe we could change it to use Selenium? In looking a bit at your
> commits, it appears I need a .java file for each test or set of tests. That
> may not be helpful for the AS-side so just being able to declare in MXML or
> AS what the expected output is and then have an engine or engines that
> compile and run in different configs and test against those outputs seems
> like it would enable more folks to write tests. I think it might be
> possible to tweak your TextButton.java to pick up its expected output from
> some external file and thus become part of that engine.
> --
> Alex Harui
> Flex SDK Team
> Adobe Systems, Inc.
> http://blogs.adobe.com/aharui
>
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
Aha! See, I'm perfectly capable of messing up a commit message ;-)
My bad, you're right, it should've read [ASJS]. Sorry for the confusion.
EdB
On Sun, Apr 14, 2013 at 8:11 AM, Alex Harui <ah...@adobe.com> wrote:
>
>
>
> On 4/13/13 1:18 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
>
>>> It isn't obvious to me that a compiler test suite needs to execute code in a
>>> runtime. But certainly, after changing the compiler and passing its Junit
>>> tests, it would be good to re-run whatever test suite we create for testing
>>> changes to the ASJS code.
>>
>> Ok, once more, as a one-liner: marmotinni is not a compiler test
>> suite, it's an JS SDK test suite. That's why I put it in the ASJS
>> repo, not in the Falcon repo...
> Aha! See, I told you I am capable of missing an important fact. Question
> though: in one commit the subject was [FalconJX], but the repo was
> flex-asjs. Did you add that tag to the commit message or is there something
> messed up with the commit emails.
>
>
> --
> Alex Harui
> Flex SDK Team
> Adobe Systems, Inc.
> http://blogs.adobe.com/aharui
>
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
RE: Meet 'marmotinni', an ASJS testing framework
Posted by Tigran Najaryan <ti...@gmail.com>.
> From: Alex Harui [mailto:aharui@adobe.com]
> OK, probably just wishful thinking on my part. I think we both agree
> that it would be great if you could write a test on AS and have it run
> in JS.
I definitely agree, that would be great, that was what I initially thought
we could do and what excited me. Alas now that I think more of it I do not
see a reasonably easy way to achieve that.
> What I was trying to point out is that in FlexJS, the JS code for
> Combobox, for example, is not generated by cross compiling the AS code
> for ComboBox.
> Instead, under the covers of the ComboBox API, we hand-wrote one that
> works on Flash, and another one that works in JS (that uses the
> HTMLSelect).
Yes, I already know this.
> We haven't implemented an API in ComboBox or one of its
> plug-ins that plays back selection of an option, but if we did, and
> called it maybe "playbackSelection", it would make the right calls in
> AS and a different set of calls in JS, so theoretically, the test
> script that calls playbackSelection would just "do the right thing".
> But this may break down in more complex scenarios.
Sure, this is technically possible, it is just not a trivial thing to do.
Essentially it means implementing a full new automation API on all FlexJS
components. I suspect it is too much effort for the current Apache Flex
team. How many committers does the Flex team currently have anyway?
> Just for my education, does the RTS file get "compiled" or is it
> interpreted by the test engine?
It is interpreted. I am actually the person who is responsible for the
interpreter and the runtime, hence my interest in FalconJS.
> And thanks for the list of things to keep in mind. That was very
> helpful.
You are welcome.
Tigran.
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
On 4/15/13 9:10 AM, "Tigran Najaryan" <ti...@gmail.com> wrote:
>> From: Alex Harui [mailto:aharui@adobe.com]
>>
>> OK, but FlexJS is essentially wrapping JS constructs and presenting
>> them in AS. The goal for the components is the present the same APIs
>> to the developer by wrapping and emulating where appropriate.
> Sorry, I probably miss something but I do not see what does this change from
> automation point of view.
OK, probably just wishful thinking on my part. I think we both agree that
it would be great if you could write a test on AS and have it run in JS.
What I was trying to point out is that in FlexJS, the JS code for Combobox,
for example, is not generated by cross compiling the AS code for ComboBox.
Instead, under the covers of the ComboBox API, we hand-wrote one that works
on Flash, and another one that works in JS (that uses the HTMLSelect). We
haven't implemented an API in ComboBox or one of its plug-ins that plays
back selection of an option, but if we did, and called it maybe
"playbackSelection", it would make the right calls in AS and a different set
of calls in JS, so theoretically, the test script that calls
playbackSelection would just "do the right thing". But this may break down
in more complex scenarios.
Just for my education, does the RTS file get "compiled" or is it interpreted
by the test engine?
And thanks for the list of things to keep in mind. That was very helpful.
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
RE: Meet 'marmotinni', an ASJS testing framework
Posted by Tigran Najaryan <ti...@gmail.com>.
> From: Alex Harui [mailto:aharui@adobe.com]
>
> OK, but FlexJS is essentially wrapping JS constructs and presenting
> them in AS. The goal for the components is the present the same APIs
> to the developer by wrapping and emulating where appropriate.
Sorry, I probably miss something but I do not see what does this change from
automation point of view.
> So, forget what the current Flex Automation API does. If you had a chance
> to design a new automation API for FlexJS where the components are
> already wrapping (that uses plug-ins), could it be done?
The current JS output produces a HTML page at runtime which can be automated
by in-browser HTML automation tools. I am not sure there is a need for
separate automation API.
I think the right approach would be to generate JS in a way that is
'friendly' to automation tools. Including the 'id' property or honoring the
'automationName' property defined in the MXML are steps in the right
direction.
Some 'best practice' points for successful HTML automation are:
1. As mentioned earlier having unique and/or descriptive property values
defined on generated DOM elements.
2. Ensuring helper HTML elements that are generated in DOM hierarchy purely
for cosmetics are clearly marked so that they can be easily ignored by the
automation tool. Ideally the automation tool must only operate on HTML
elements which are functional part of the UI and can be directly interacted
by the human user.
3. Ensuring that descriptive properties remain constant across builds and
during runtime. The autogenerated ids which keep changing from run to run or
from build to build are exactly what must NOT be done and which are very
undesirable from automation point of view. JS libraries such as ExtJS are
known for autogenerating the ids and are very unfriendly for automation. We
spent considerable efforts to teach RIATest to ignore autogenerated ids in
ExtJS, it is a lot more desirable to make sure the JS output does not do
this for Flex.
One other reason why I think there is no need for separate automation API is
that it may require significant efforts to develop and maintain (e.g. the
Flex Automation Framework is currently about 62K lines of source code).
I will keep thinking about the automation aspects of FlexJS and will post
back if I have anything useful to say.
Tigran.
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
On 4/15/13 12:39 AM, "Tigran Najaryan" <ti...@gmail.com> wrote:
>> OK, but given that we trans-compile AS to JS and your test language is ES-
>> like, there is no way to wrap things so they work?
>
> I don't think so. The problem is that tests on Flex apps compiled to SWF use
> action names specific to each component (usually corresponding to
> appropriate Flex event names) as defined by Flex Automation framework. For
> example to open a ComboBox and select an item you do something like this in
> RIATest:
OK, but FlexJS is essentially wrapping JS constructs and presenting them in
AS. The goal for the components is the present the same APIs to the
developer by wrapping and emulating where appropriate. So, forget what the
current Flex Automation API does. If you had a chance to design a new
automation API for FlexJS where the components are already wrapping (that
uses plug-ins), could it be done?
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
RE: Meet 'marmotinni', an ASJS testing framework
Posted by Tigran Najaryan <ti...@gmail.com>.
> OK, but given that we trans-compile AS to JS and your test language is ES-
> like, there is no way to wrap things so they work?
I don't think so. The problem is that tests on Flex apps compiled to SWF use
action names specific to each component (usually corresponding to
appropriate Flex event names) as defined by Flex Automation framework. For
example to open a ComboBox and select an item you do something like this in
RIATest:
FlexComboBox("myCB").open();
FlexComboBox("myCB ").select("Some item");
The open() and select() actions are handled by the automation delegate for
mx.controls.ComboBox component.
On the contrary the same application compiled to JS will not only know
nothing about FlexComboBox (which could be solved by my earlier suggestion),
it also does not use open() and select() actions, it uses either low-level
mouse events dispatched directly to HTML DOM elements, or HTML element
specific events. In this case the correct action will be change():
HtmlSelect("myCB").change("Some item");
Note that we do not even have an action corresponding to open(). We could do
click() but it does not have exact same effect.
In some other cases a sequence of Flex-specific actions can be replaced by
mouse events. For example Flex Automation Framework uses itemOpen() action
to expand an item of mx.controls.Tree, however this must be likely
simulated using simple click() action on the relevant DOM element in the JS
version of the test.
So, generally there is no match between the actions that one needs to
simulate on the SWF output and the actions that must be simulated on the JS
output to have the exact same effect. The names of the actions, the
parameters and the sequence, everything can be different. This unfortunately
means it is not possible to have a single uniform test script that can act
on both output versions the same way.
Tigran.
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
On 4/14/13 11:30 PM, "Tigran Najaryan" <ti...@gmail.com> wrote:
>> If we go a step further and also emit Flex class name in another property
>> (perhaps in the 'class' property of DOM elements) we could make RIATest
>> automatically recognize which Flex component does each DOM element
>> represent and even aim to have a universal syntax for test scripts that
> can
>> be run against both SWF target and JS target of the same Flex application.
>>
>> What this basically means is that you can write a test script on a SWF
> target
>> and then make sure the same script runs exactly the same way on the JS
>> target. That coupled with the ability to run the tests on multiple
> browsers
>> and 2 operating systems will give a very powerful way to validate the JS
>> output and ensure it works exactly the same way as the SWF output of the
>> same Flex app. This will require modifications to RIATest but the end
> result
>> will be really exciting so I think we can consider adding such capability
> to an
>> upcoming RIATest version.
>
> Sorry, I was too quick. This is not going to work. The actions on SWF and
> HTML objects are done in different terms so there is no way to have
> identical tests. Ignore this particular idea.
>
OK, but given that we trans-compile AS to JS and your test language is
ES-like, there is no way to wrap things so they work?
> Tigran.
>
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
RE: Meet 'marmotinni', an ASJS testing framework
Posted by Tigran Najaryan <ti...@gmail.com>.
> If we go a step further and also emit Flex class name in another property
> (perhaps in the 'class' property of DOM elements) we could make RIATest
> automatically recognize which Flex component does each DOM element
> represent and even aim to have a universal syntax for test scripts that
can
> be run against both SWF target and JS target of the same Flex application.
>
> What this basically means is that you can write a test script on a SWF
target
> and then make sure the same script runs exactly the same way on the JS
> target. That coupled with the ability to run the tests on multiple
browsers
> and 2 operating systems will give a very powerful way to validate the JS
> output and ensure it works exactly the same way as the SWF output of the
> same Flex app. This will require modifications to RIATest but the end
result
> will be really exciting so I think we can consider adding such capability
to an
> upcoming RIATest version.
Sorry, I was too quick. This is not going to work. The actions on SWF and
HTML objects are done in different terms so there is no way to have
identical tests. Ignore this particular idea.
Tigran.
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
On 4/13/13 1:18 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
>> It isn't obvious to me that a compiler test suite needs to execute code in a
>> runtime. But certainly, after changing the compiler and passing its Junit
>> tests, it would be good to re-run whatever test suite we create for testing
>> changes to the ASJS code.
>
> Ok, once more, as a one-liner: marmotinni is not a compiler test
> suite, it's an JS SDK test suite. That's why I put it in the ASJS
> repo, not in the Falcon repo...
Aha! See, I told you I am capable of missing an important fact. Question
though: in one commit the subject was [FalconJX], but the repo was
flex-asjs. Did you add that tag to the commit message or is there something
messed up with the commit emails.
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
> It isn't obvious to me that a compiler test suite needs to execute code in a
> runtime. But certainly, after changing the compiler and passing its Junit
> tests, it would be good to re-run whatever test suite we create for testing
> changes to the ASJS code.
Ok, once more, as a one-liner: marmotinni is not a compiler test
suite, it's an JS SDK test suite. That's why I put it in the ASJS
repo, not in the Falcon repo...
EdB
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Markus Gritsch <ma...@gmx.de>.
Hi Erik,
I don`t know what marmotinni is doing at source code level but from your comments it`s sound similar to gorillalogic`s monkeytalk. Monkeytalk has Agents for every major browser, desktop, mobile and supports testing mobile native apps as well. Its open source, implemented in Java. check it out: https://www.gorillalogic.com/developer-resources/downloads
Regards,
Markus
MILLIPEDE LABS - PROJECT MILLIPEDE
"PRIVACY PROTECTION" - FOR everyONE - meets SOCIAL NETWORKS
Straubinger Str. 19
9363 Oberschneiding
Germany
Telefon: +49 9426 85 260 140
Web: millipede.me
On Apr 12, 2013, at 9:28 PM, Erik de Bruin wrote:
>> Point of order: why do you seem so opposed to this contribution? I've
>> put a lot of effort in it, and it serves a function that I don't see
>> any of the other tools being able to without serious modification...
>> It's not like this project is drowning in active contributors, so why
>> look a gift horse in the mouth?
>
> Yes... please ignore that. It's been a really long week. Sorry :-(
>
> EdB
>
>
>
> --
> Ix Multimedia Software
>
> Jan Luykenstraat 27
> 3521 VB Utrecht
>
> T. 06-51952295
> I. www.ixsoftware.nl
RE: Meet 'marmotinni', an ASJS testing framework
Posted by Tigran Najaryan <ti...@gmail.com>.
> Awesome! We'd have to figure out how to manage a licence for an entire
> project. We can't have 'a few' licences to specific people, as anyone who
is
> interested should be able to run the tests. We'd have to have an 'eternal
> licence' of some sort that would ensure that if we commit to using
RIATest,
> the licensing would never become an issue.
Yes, I realize that. We'd probably need to provide a permanent non-expiring
special license to any Apache Flex user to run the tests. A kind of
perpetual non-named "Runner" license would do this. We already have Runner
licenses which only allow to run the tests.
The Runner licenses we currently sell are named (licensed to a particular
person or organization). However I think we can do a special license type
for Apache Flex project so that any person running the Flex SDK build is
licensed to run RIATest tests that are part of the SDK Build procedure. That
should cover the legal aspects.
In additional to that we could donate several Professional edition licenses.
These allow for full usage of RIATest including developing the tests. These
licenses will be only needed by people who are actually developing tests, so
there won't be needed a lot of this type.
> Maybe you can prepare some examples specific to ASJS so we can better
> evaluate what it could do for this project. Would you be interested in
> providing those?
Sure, why not. How about I prepare a sample test for FlexJSTest_again
example?
Tigran.
RE: Meet 'marmotinni', an ASJS testing framework
Posted by Tigran Najaryan <ti...@gmail.com>.
> Hi Tigran,
>
> Thank you for your offer of a special license of RIATest for Apache
> Flex.
> I'm not familiar with RIATest at all. It appears to be more like what
> I would call an application testing framework like QTP. Is that
> correct?
Correct. RIATest does functional testing of GUI just like QTP does.
> One of the reasons Adobe wrote Mustella was to get exact
> control over the timing of frame events relative to other faked mouse
> and keyboard events. Can RIATest do something like that? We may still
> need that for testing the AS-side of the framework.
No, RIATest cannot do that, so it is certainly not a replacement for
Mustella. RIATest simulates user actions and is not aware of frame timing.
For Flex applications it uses Flex Automation framework to simulate the
actions. For HTML pages it simulates DOM events directly on HTML DOM
elements. It can also simulate system-level mouse and keyboard events on any
Windows GUI object if needed.
> Regardless, I am still interested in RIATest in two ways:
> 1) RIATest as the way to test sample apps.
I believe that would be the use for RIATest. It can simulate actions on
sample Flex applications compiled to SWF and HTML/JS targets loaded in a
browser (Chrome, Firefox and IE) to ensure they are functioning as expected.
> 2) I would love to see what changes we would need to make to FlexJS to
> get RIATest to work in our workflow so that folks can use RIATest to
> test their applications.
Actually strictly speaking no changes are mandatory. See sample test I
attached to https://issues.apache.org/jira/browse/FLEX-33489 and which tests
the output of FlexTestAgain example compiled with current FalconJS compiler.
However see below for desirable changes.
> I was not directly involved in the QTP automation capability of the
> current Flex SDK, but I think there were two main complaints about the
> way Adobe implemented it:
> A) Lots of automation code is baked into the framework whether you use
> it or not
> B) You have to choose at compile time whether to add in the rest of the
> automation code.
Correct. You need to compile the application with a few SWCs that contain
automation delegate classes for all Flex components and the core Flex
Automation framework classes. But see below on another option.
> I'm hoping that the "beads" plug-in model will solve these two issues,
> but I can't see myself having time to actually try it any time soon.
> It appears that RIATest doesn't require additional properties on the
> component (other
> than the id property you mentioned). Does RIATest require other
> properties
> on the components?
RIATest does not require additional properties and it is possible to write
tests on objects which are missing ids however having uniquely identifying
and descriptive text assigned to the 'id' or other publicly accessible
property greatly simplifies test automation and makes test scripts more
descriptive and less sensitive to application changes. This is generally
true for any GUI test automation not just for RIATest. RIATest scripts can
use any public property (or a combination of properties) to identify the
acting object however the 'id' property when present is used by default for
the HTML elements.
Flex Automation framework also supports 'automationName' and
'automationValue' property. You can either explicitly set values of these
properties in your MXML or ActionScript for all UI components derived from
UIComponent. Alternatively 'automationName' or 'automationValue' will be
derived from other public properties such as 'text' or 'label'. The
'automationName' is the default property that is used by RIATest to identify
Flex objects during tests (although it can use any combination of public
properties and it is possible to perform strict or regex matching).
It would be great if FalconJS honored 'automationName' and emitted
corresponding 'data-automation-name' property on DOM elements.
If we go a step further and also emit Flex class name in another property
(perhaps in the 'class' property of DOM elements) we could make RIATest
automatically recognize which Flex component does each DOM element represent
and even aim to have a universal syntax for test scripts that can be run
against both SWF target and JS target of the same Flex application.
What this basically means is that you can write a test script on a SWF
target and then make sure the same script runs exactly the same way on the
JS target. That coupled with the ability to run the tests on multiple
browsers and 2 operating systems will give a very powerful way to validate
the JS output and ensure it works exactly the same way as the SWF output of
the same Flex app. This will require modifications to RIATest but the end
result will be really exciting so I think we can consider adding such
capability to an upcoming RIATest version.
> Does it run against a production SWF so it doesn't
> have issue B?
It doesn't have that issue. One of the options in RIATest is running against
a production SWF, so it is not necessary to make the application ready for
automation at compile time. Technically RIATest first loads a trivial
application containing one full screen SWFLoader component and compiled with
necessary automation libraries. This SWFLoader in turn loads the end user's
SWF application to test. This way the loaded SWF application becomes fully
exposed to Flex Automation framework and testable by RIATest. This option
can be used as long as the Flex application is loadable by SWFLoader and
does not change its behavior when loaded in a SWFLoader. I believe this will
be true for the majority of trivial sample applications that will be part of
the test bed.
A couple more pros for using RIATest:
The scripts in RIATest are written in simplified ECMAScript with minor
syntax extensions so it is a good match for Flex devs who already know
ActionScript.
RIATest can also be invoked from Hudson/Jenkins and generates JUnit
compatible results allowing to integrate it with continuous builds.
Thanks
Tigran.
P.S. Sorry for a long email, I can't stop when I start talking about
RIATest. :-)
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
Yes, I encountered this myself when writing marmotinni. FlexJS needs
to add some ID to the nodes it creates in the DOM.
EdB
On Sat, Apr 13, 2013 at 2:19 PM, Tigran Najaryan <ti...@gmail.com> wrote:
>> Maybe you can prepare some examples specific to ASJS so we can better
>> evaluate what it could do for this project. Would you be interested in
>> providing those?
>
> I posted a small sample RIATest project that tests FlexJSTest_again example
> on Jira: https://issues.apache.org/jira/browse/FLEX-33489
>
> Please see the README file in the zip.
>
>
> An additional note:
>
> Currently generated HTML DOM is not ideal from test automation point of
> view. It would be great if FalconJS could emit additional information such
> id from the MXML (the id could go to the "id" of HTML element or to
> data-automation-id). This will help with automation and will earn you a lot
> of bonus points from test engineers.
>
> If you have any questions feel free to email to this list or directly to my
> email.
>
> Tigran.
>
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
Hi Tigran,
Thank you for your offer of a special license of RIATest for Apache Flex.
I'm not familiar with RIATest at all. It appears to be more like what I
would call an application testing framework like QTP. Is that correct? One
of the reasons Adobe wrote Mustella was to get exact control over the timing
of frame events relative to other faked mouse and keyboard events. Can
RIATest do something like that? We may still need that for testing the
AS-side of the framework.
Regardless, I am still interested in RIATest in two ways:
1) RIATest as the way to test sample apps.
2) I would love to see what changes we would need to make to FlexJS to get
RIATest to work in our workflow so that folks can use RIATest to test their
applications.
I was not directly involved in the QTP automation capability of the current
Flex SDK, but I think there were two main complaints about the way Adobe
implemented it:
A) Lots of automation code is baked into the framework whether you use it or
not
B) You have to choose at compile time whether to add in the rest of the
automation code.
I'm hoping that the "beads" plug-in model will solve these two issues, but I
can't see myself having time to actually try it any time soon. It appears
that RIATest doesn't require additional properties on the component (other
than the id property you mentioned). Does RIATest require other properties
on the components? Does it run against a production SWF so it doesn't have
issue B?
Thanks,
-Alex
On 4/13/13 5:19 AM, "Tigran Najaryan" <ti...@gmail.com> wrote:
>> Maybe you can prepare some examples specific to ASJS so we can better
>> evaluate what it could do for this project. Would you be interested in
>> providing those?
>
> I posted a small sample RIATest project that tests FlexJSTest_again example
> on Jira: https://issues.apache.org/jira/browse/FLEX-33489
>
> Please see the README file in the zip.
>
>
> An additional note:
>
> Currently generated HTML DOM is not ideal from test automation point of
> view. It would be great if FalconJS could emit additional information such
> id from the MXML (the id could go to the "id" of HTML element or to
> data-automation-id). This will help with automation and will earn you a lot
> of bonus points from test engineers.
>
> If you have any questions feel free to email to this list or directly to my
> email.
>
> Tigran.
>
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
RE: Meet 'marmotinni', an ASJS testing framework
Posted by Tigran Najaryan <ti...@gmail.com>.
> Maybe you can prepare some examples specific to ASJS so we can better
> evaluate what it could do for this project. Would you be interested in
> providing those?
I posted a small sample RIATest project that tests FlexJSTest_again example
on Jira: https://issues.apache.org/jira/browse/FLEX-33489
Please see the README file in the zip.
An additional note:
Currently generated HTML DOM is not ideal from test automation point of
view. It would be great if FalconJS could emit additional information such
id from the MXML (the id could go to the "id" of HTML element or to
data-automation-id). This will help with automation and will earn you a lot
of bonus points from test engineers.
If you have any questions feel free to email to this list or directly to my
email.
Tigran.
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
Tigran,
Awesome! We'd have to figure out how to manage a licence for an entire
project. We can't have 'a few' licences to specific people, as anyone
who is interested should be able to run the tests. We'd have to have
an 'eternal licence' of some sort that would ensure that if we commit
to using RIATest, the licensing would never become an issue.
Maybe you can prepare some examples specific to ASJS so we can better
evaluate what it could do for this project. Would you be interested in
providing those?
EdB
On Sat, Apr 13, 2013 at 8:37 AM, Tigran Najaryan <ti...@gmail.com> wrote:
> Guys, if you intend to do functional GUI testing my company will be happy to
> donate a few licenses of RIATest (http://www.cogitek.com/riatest.html) to
> Apache Flex. It can test both Flex and HTML in a browser, works on Windows
> and Mac. It is a commercial tool but I think we can make it free for Flex
> SDK testing.
>
> Adobe Professional Services were using RIATest internally so there must be
> something good in the tool. :-)
>
> I have lots of experience with this stuff and will be happy to personally
> lead the efforts initially and help anyone interested to get up to speed
> with RIATest.
>
> Let me know if this is interesting and I will look further into what can be
> done with licenses.
>
> Tigran.
>
>
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
RE: Meet 'marmotinni', an ASJS testing framework
Posted by Tigran Najaryan <ti...@gmail.com>.
Guys, if you intend to do functional GUI testing my company will be happy to
donate a few licenses of RIATest (http://www.cogitek.com/riatest.html) to
Apache Flex. It can test both Flex and HTML in a browser, works on Windows
and Mac. It is a commercial tool but I think we can make it free for Flex
SDK testing.
Adobe Professional Services were using RIATest internally so there must be
something good in the tool. :-)
I have lots of experience with this stuff and will be happy to personally
lead the efforts initially and help anyone interested to get up to speed
with RIATest.
Let me know if this is interesting and I will look further into what can be
done with licenses.
Tigran.
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
On 4/12/13 12:28 PM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
>> Point of order: why do you seem so opposed to this contribution? I've
>> put a lot of effort in it, and it serves a function that I don't see
>> any of the other tools being able to without serious modification...
>> It's not like this project is drowning in active contributors, so why
>> look a gift horse in the mouth?
>
> Yes... please ignore that. It's been a really long week. Sorry :-(
>
OK, I will. I'm sorry if I'm coming across as a negative person. I
definitely appreciate your efforts especially since you are doing it in your
spare time.
That said, I think I should be able to raise questions about your commits.
I expect you and the rest of the community to challenge anything I do. I am
certainly capable of not being aware of some important fact.
On this particular subject, I am wary about "another" test framework. I was
hoping when FlexUnit arrived we would be able to leverage it and maybe parts
of Mustella to test the AS and JS code. Mustella already knows how to
launch a browser (it doesn't have to launch a standalone player). In
thinking ahead several years, if FlexJS grows to do 80% of what Flex can
today, I would not want to have to rewrite all of those mustella tests, so
it would be great to be able to use them.
It isn't obvious to me that a compiler test suite needs to execute code in a
runtime. But certainly, after changing the compiler and passing its Junit
tests, it would be good to re-run whatever test suite we create for testing
changes to the ASJS code.
But again, these are just "concerns" and not a veto or firm opposition. And
again, thanks for all the work you've been doing.
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
> Point of order: why do you seem so opposed to this contribution? I've
> put a lot of effort in it, and it serves a function that I don't see
> any of the other tools being able to without serious modification...
> It's not like this project is drowning in active contributors, so why
> look a gift horse in the mouth?
Yes... please ignore that. It's been a really long week. Sorry :-(
EdB
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
RE: Meet 'marmotinni', an ASJS testing framework
Posted by "Michael A. Labriola" <la...@digitalprimates.net>.
>If FlexUnit can test the functioning of an application (i.e. check the value of a label after it has faked a click on a button, triggering an event handler), then I'm all >for it. Maybe Mike can chime in on this?
FlexUnit does Unit and Integration tests. Integration allow some of those things (like check a label after it faked a button click) but it's really intended to test things on a smaller level. Think a class at a time.
Many moons ago, Alex and I talked over whether we could run mustella style tests on top of the FlexUnit core runner. Therefore having a unified set of unit, integration and functional tests. There are some challenges if we were to proceed, namely that mustella actually launches player (n) times whereas FlexUnit was more about being unit and integration testing, so it runs within the confines of a single launch presently.
Regarding this, if someone wanted to do the work, I am sure this could be made to work, however, that would still all be the ActionScript side of things, not the output JavaScript.
Mike
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
> Yeah, I meant functional testing. IMHO, these are functional tests. They
> test that the compiler is functioning as a whole. It isn't testing the
> individual units like a set of tests for the JSEmitter.java class. And a
> separate suite of functional tests for ASJS would probably live in the asjs
> repo, not the falcon repo. But I could certainly be wrong here.
That's part of the confusion. Marmotinni is not designed for unit
testing the compiler (FalconFX), that's what 'compiler.js.tests' is
for. It's designed to test if an application that is designed to do
one thing when it's playing in the Flash Player does the same thing
when it's playing as an JS application in all of the major browsers.
It checks the functionality of an application: functional testing.
>> That would be ideal, and maybe we can use the format that mustella
>> uses (MXML if I'm correct?) and make marmotinni use the same format,
>> making if possible to use the same functional tests for both the AS
>> (Flash Player) side and the JS (browser) side.
> IMO, MXML or AS. We're in the business of translating either to JS aren't
> we?
Not what I was saying. Mustella uses MXML as the language to define
the tests. I'm saying that we might want to use a similar language to
drive a combined AS/JS application functional application test
framework.
> Yeah, but it seems like I have to override by copying the entire test
> including its source instead of having a set of source inputs and an
> overridable set of expected outputs. That's what I was thinking of
> refactoring.
That sounds like a great plan, but it doesn't have any bearing on the
current topic ;-)
> Well, we own the code for FlexUnit, so at the point it takes the compiler
> output and launches the player and tears it down (I am just assuming it does
> that) maybe we could change it to use Selenium? In looking a bit at your
> commits, it appears I need a .java file for each test or set of tests. That
> may not be helpful for the AS-side so just being able to declare in MXML or
> AS what the expected output is and then have an engine or engines that
> compile and run in different configs and test against those outputs seems
> like it would enable more folks to write tests. I think it might be
> possible to tweak your TextButton.java to pick up its expected output from
> some external file and thus become part of that engine.
If FlexUnit can test the functioning of an application (i.e. check the
value of a label after it has faked a click on a button, triggering an
event handler), then I'm all for it. Maybe Mike can chime in on this?
Point of order: why do you seem so opposed to this contribution? I've
put a lot of effort in it, and it serves a function that I don't see
any of the other tools being able to without serious modification...
It's not like this project is drowning in active contributors, so why
look a gift horse in the mouth?
EdB
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
On 4/12/13 11:12 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
>> I don't know much about testing frameworks other than mustella, so I'm
>> mostly asking dumb questions. I don't have a fully-formed plan on how to
>> test all of this stuff, so I'm not saying you made the wrong decision. I'm
>> just curious. How do I test the AS Side? Does this mean I have two
>> versions of each test?
>
> With this solution, you don't test the AS side (yet). Unless we write
> a meta-layer that is interpreted by both an AS functional testing
> framework (mustella?) and a JS functional testing framework
> (marmotinni?), there is no framework Google knows about that does
> functional testing on both AS and JS based on the same test data/file.
I would like to explore what that layer looks like before we author a bunch
of tests that have to be duplicated to test the AS side. I still think
FlexUnit can do functional testing just like Junit can, but I could be
wrong.
>
>> I was just looking at the compiler.jx.tests trying to figure out where to
>> add more tests. These seem like feature tests to me, do you agree? Yet we
>> are using Junit to run them. I haven't used FlexUnit but I assume it can do
>> unit and functional testing just like Junit. And we own the code so we can
>> make it do that if needed.
>
> Feature tests? If by that you mean functional tests, then no, I don't
> agree. The tests in compiler.jx.tests are unit tests that test if the
> output string from FalconJX matches the expected value, given a
> certain input. It in no way tests whether the input or the output
> actually DOES anything. I doesn't check if given the code for a click
> on a button, the code correctly fires an event that causes the text of
> a specified label to change to the correct value. That is functional
> testing, IMHO.
Yeah, I meant functional testing. IMHO, these are functional tests. They
test that the compiler is functioning as a whole. It isn't testing the
individual units like a set of tests for the JSEmitter.java class. And a
separate suite of functional tests for ASJS would probably live in the asjs
repo, not the falcon repo. But I could certainly be wrong here.
>
>> I do think I just want to write the "source" for a test once and then have
>> it run "everywhere".
>
> That would be ideal, and maybe we can use the format that mustella
> uses (MXML if I'm correct?) and make marmotinni use the same format,
> making if possible to use the same functional tests for both the AS
> (Flash Player) side and the JS (browser) side.
IMO, MXML or AS. We're in the business of translating either to JS aren't
we?
>
>> As I've been going through the compiler.jx.tests, it makes me want to
>> refactor those tests as well so they can be used to test Falcon's AS
>> handling too.
>
> The tests in 'org.apache.flex.compiler.internal.codegen.as' test
> exactly that. They test AS to AS compilation of most language features
> (some work needs to be done writing tests for the missing feature).
> All other ASJS test 'suites' (amd, goog, flexjs) subclass these tests
> and only override them where the expected output differs from that of
> the parent. It was set up this way to ensure that starting with
> testing all possible AS output, all the language features would also
> be tested in the subclasses, i.e. the various JS output formats.
Yeah, but it seems like I have to override by copying the entire test
including its source instead of having a set of source inputs and an
overridable set of expected outputs. That's what I was thinking of
refactoring.
>
>> At one point several years ago, Mike L and I discussed authoring FlexUnit
>> tests in MXML. I still like the idea as it more tightly controls the way
>> tests are authored.
>
> As I mentioned above, we might look into mustella's MXML formatted
> tests and see if we can leverage that in marmotinni. I don't think
> FlexUnit (being a unit testing framework) will be able to run
> functional tests in both the Flash Player and the various browsers...
Well, we own the code for FlexUnit, so at the point it takes the compiler
output and launches the player and tears it down (I am just assuming it does
that) maybe we could change it to use Selenium? In looking a bit at your
commits, it appears I need a .java file for each test or set of tests. That
may not be helpful for the AS-side so just being able to declare in MXML or
AS what the expected output is and then have an engine or engines that
compile and run in different configs and test against those outputs seems
like it would enable more folks to write tests. I think it might be
possible to tweak your TextButton.java to pick up its expected output from
some external file and thus become part of that engine.
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
> I don't know much about testing frameworks other than mustella, so I'm
> mostly asking dumb questions. I don't have a fully-formed plan on how to
> test all of this stuff, so I'm not saying you made the wrong decision. I'm
> just curious. How do I test the AS Side? Does this mean I have two
> versions of each test?
With this solution, you don't test the AS side (yet). Unless we write
a meta-layer that is interpreted by both an AS functional testing
framework (mustella?) and a JS functional testing framework
(marmotinni?), there is no framework Google knows about that does
functional testing on both AS and JS based on the same test data/file.
> I was just looking at the compiler.jx.tests trying to figure out where to
> add more tests. These seem like feature tests to me, do you agree? Yet we
> are using Junit to run them. I haven't used FlexUnit but I assume it can do
> unit and functional testing just like Junit. And we own the code so we can
> make it do that if needed.
Feature tests? If by that you mean functional tests, then no, I don't
agree. The tests in compiler.jx.tests are unit tests that test if the
output string from FalconJX matches the expected value, given a
certain input. It in no way tests whether the input or the output
actually DOES anything. I doesn't check if given the code for a click
on a button, the code correctly fires an event that causes the text of
a specified label to change to the correct value. That is functional
testing, IMHO.
> I do think I just want to write the "source" for a test once and then have
> it run "everywhere".
That would be ideal, and maybe we can use the format that mustella
uses (MXML if I'm correct?) and make marmotinni use the same format,
making if possible to use the same functional tests for both the AS
(Flash Player) side and the JS (browser) side.
> As I've been going through the compiler.jx.tests, it makes me want to
> refactor those tests as well so they can be used to test Falcon's AS
> handling too.
The tests in 'org.apache.flex.compiler.internal.codegen.as' test
exactly that. They test AS to AS compilation of most language features
(some work needs to be done writing tests for the missing feature).
All other ASJS test 'suites' (amd, goog, flexjs) subclass these tests
and only override them where the expected output differs from that of
the parent. It was set up this way to ensure that starting with
testing all possible AS output, all the language features would also
be tested in the subclasses, i.e. the various JS output formats.
> At one point several years ago, Mike L and I discussed authoring FlexUnit
> tests in MXML. I still like the idea as it more tightly controls the way
> tests are authored.
As I mentioned above, we might look into mustella's MXML formatted
tests and see if we can leverage that in marmotinni. I don't think
FlexUnit (being a unit testing framework) will be able to run
functional tests in both the Flash Player and the various browsers...
EdB
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
On 4/12/13 9:45 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
> Ok, maybe I didn't explain the idea correctly: marmotinni is meant as
> a testing framework for the HTML/JS output of an ASJS framework
> (currently focussed on FlexJS). We do want a way to automatically test
> the functionality of the FalconJX output (debug and release) across
> the various browsers, so we can be sure that whatever we do to the
> various frameworks or FalconJX doesn't adversely affect it, don't we?
Yup.
>
> Why would you want to roll our own, based on a framework that wasn't
> designed for it, when I give you this solution?
>
I don't know much about testing frameworks other than mustella, so I'm
mostly asking dumb questions. I don't have a fully-formed plan on how to
test all of this stuff, so I'm not saying you made the wrong decision. I'm
just curious. How do I test the AS Side? Does this mean I have two
versions of each test?
I was just looking at the compiler.jx.tests trying to figure out where to
add more tests. These seem like feature tests to me, do you agree? Yet we
are using Junit to run them. I haven't used FlexUnit but I assume it can do
unit and functional testing just like Junit. And we own the code so we can
make it do that if needed.
And I have no clue if it would be better/faster for us to modify FlexUnit to
cover JS unit and functional testing, or give it some way to leverage what
you did to run it on Selenium. But I was considering whether there are
extensive suites of FlexUnit tests out there that folks may want to leverage
as FlexJS grows up.
I do think I just want to write the "source" for a test once and then have
it run "everywhere".
As I've been going through the compiler.jx.tests, it makes me want to
refactor those tests as well so they can be used to test Falcon's AS
handling too.
At one point several years ago, Mike L and I discussed authoring FlexUnit
tests in MXML. I still like the idea as it more tightly controls the way
tests are authored.
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
Ok, maybe I didn't explain the idea correctly: marmotinni is meant as
a testing framework for the HTML/JS output of an ASJS framework
(currently focussed on FlexJS). We do want a way to automatically test
the functionality of the FalconJX output (debug and release) across
the various browsers, so we can be sure that whatever we do to the
various frameworks or FalconJX doesn't adversely affect it, don't we?
Why would you want to roll our own, based on a framework that wasn't
designed for it, when I give you this solution?
EdB
On Fri, Apr 12, 2013 at 6:32 PM, Alex Harui <ah...@adobe.com> wrote:
>
>
>
> On 4/12/13 9:18 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
>
>> FlexUnit does functional HTML testing in various browsers?
> OK, I didn't realize you were focusing on Functional Testing.
>
> I have never used FlexUnit myself (in fact, I need to make sure it doesn't
> have direct ties to the old Flex SDK), but the code has been donated to
> Apache Flex and I think it can do some sort of functional testing. Maybe it
> should be the basis of a JS testing framework. I'm imagining having a set
> of FlexUnit tests written to test the AS side of the framework on Flash. I
> would like to know if we can re-use these tests to run the same tests on the
> JS side.
>
>
>
> --
> Alex Harui
> Flex SDK Team
> Adobe Systems, Inc.
> http://blogs.adobe.com/aharui
>
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
On 4/12/13 9:18 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
> FlexUnit does functional HTML testing in various browsers?
OK, I didn't realize you were focusing on Functional Testing.
I have never used FlexUnit myself (in fact, I need to make sure it doesn't
have direct ties to the old Flex SDK), but the code has been donated to
Apache Flex and I think it can do some sort of functional testing. Maybe it
should be the basis of a JS testing framework. I'm imagining having a set
of FlexUnit tests written to test the AS side of the framework on Flash. I
would like to know if we can re-use these tests to run the same tests on the
JS side.
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Erik de Bruin <er...@ixsoftware.nl>.
FlexUnit does functional HTML testing in various browsers? If that is
the case I've completely overlooked it and wasted a bunch of my time
:-(
EdB
On Fri, Apr 12, 2013 at 6:14 PM, Alex Harui <ah...@adobe.com> wrote:
> I haven't really looked yet, but why aren't we going to use FlexUnit?
>
>
> On 4/12/13 7:02 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
>
>> Hi,
>>
>> I've just committed a new testing framework to the ASJS repo. This
>> testing framework, which I've called 'marmotinni', following the
>> 'mustella' convention (Google it ;-), consists of FalconJX, asjs
>> projects, Java test classes (using the Selenium browser testing
>> classes) and an ant script to tie them all together.
>>
>> Right now there's only one test. It consists of a FlexJS project that
>> displays a single TextButton 100px from the left side of the browser
>> window. The ant script builds a JS release version using FalconJX. The
>> 'TextButton' Java class uses the Selenium framework to display the
>> FlexJS release build in a Firefox window, see if the button is there
>> and in the correct position. The result of the test is passed back to
>> ant and all artefacts are cleaned up.
>>
>> Please take a look and let me know what you think.
>>
>> EdB
>>
>>
>>
>> --
>> Ix Multimedia Software
>>
>> Jan Luykenstraat 27
>> 3521 VB Utrecht
>>
>> T. 06-51952295
>> I. www.ixsoftware.nl
>
> --
> Alex Harui
> Flex SDK Team
> Adobe Systems, Inc.
> http://blogs.adobe.com/aharui
>
--
Ix Multimedia Software
Jan Luykenstraat 27
3521 VB Utrecht
T. 06-51952295
I. www.ixsoftware.nl
Re: Meet 'marmotinni', an ASJS testing framework
Posted by Alex Harui <ah...@adobe.com>.
I haven't really looked yet, but why aren't we going to use FlexUnit?
On 4/12/13 7:02 AM, "Erik de Bruin" <er...@ixsoftware.nl> wrote:
> Hi,
>
> I've just committed a new testing framework to the ASJS repo. This
> testing framework, which I've called 'marmotinni', following the
> 'mustella' convention (Google it ;-), consists of FalconJX, asjs
> projects, Java test classes (using the Selenium browser testing
> classes) and an ant script to tie them all together.
>
> Right now there's only one test. It consists of a FlexJS project that
> displays a single TextButton 100px from the left side of the browser
> window. The ant script builds a JS release version using FalconJX. The
> 'TextButton' Java class uses the Selenium framework to display the
> FlexJS release build in a Firefox window, see if the button is there
> and in the correct position. The result of the test is passed back to
> ant and all artefacts are cleaned up.
>
> Please take a look and let me know what you think.
>
> EdB
>
>
>
> --
> Ix Multimedia Software
>
> Jan Luykenstraat 27
> 3521 VB Utrecht
>
> T. 06-51952295
> I. www.ixsoftware.nl
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui