You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@royale.apache.org by Harbs <ha...@gmail.com> on 2017/11/03 11:19:47 UTC
Unit Tests et. al.
One topic which keeps coming up is better test coverage for Royale.
I think this is becoming a critical issue for a few reasons:
1. As we get close to version 1.0 it’s necessary to have good test coverage for confidence of quality and confidence that we don’t introduce recessive bugs.
2. It’s really hard to accept Github pull requests without examining the code VERY well that it does not introduce recessive bugs. CI which runs automated tests could give a preliminary test on pull requests to ensure that they don’t break anything. If the pull requests do break something, it allows the requester to fix the problem with confidence without taking others’ time.
I think we need to break up testing into pieces and figure out a strategy to implement automated tests in a way that they are maintainable.
Some points:
1. I think integration into something like Travis would be very helpful.
2. I think there’s a Jenkins plugin for building pull requests. Not sure how useful it is.[1]
3. Josh has created a Node.js-compatible test-runner architecture which could be useful for unit tests on parts of the framework which don’t rely on browser features. (i.e. models and the like) He mentioned the possibility of donating it, and I think that would be a very useful feature.
4. For UI and integration tests, there seem to be some pretty cool integrations using Selenium.[2][3]
5. I think the main testing effort needs to be using JS and something like Josh’s testing framework for non-UI pieces and some easy-to-use Selenium integration for visual pieces and integration tests.
6. We probably also want some API endpoints we can test off of for integration testing.
I’m willing to invest time into this.
It’s going to be a lot of work building out the tests and I think we need a plan for that. My thoughts:
1. Step one is to make it easy to write meaningful automated tests and establish a clear pattern.
2. Step two is to start writing tests starting from the most-used/easiest to beak pieces and work out from there.
3. Once the pattern is established, any new pieces MUST have testing coverage.
4. When fixing bugs, attention should be paid to adding testing for that component.
5. When a pull request comes in on a piece which does not have unit test, a test must be written before accepting the pull request. The test does not need to be written by the requester. Before examining the request, the test should be written to pass for expected behavior and fail for the bug that the pull request is attempting to fix (assuming the pull request is to fix a bug).
Thoughts?
Harbs
P.S. I’m thinking of coming to the US in late December/early January. I would be interested in getting together for a hacking session with folks who are available.
[1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin <https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin>
[2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>
[3]https://saucelabs.com/products/integrations <https://saucelabs.com/products/integrations>
Re: Unit Tests et. al.
Posted by Josh Tynjala <jo...@apache.org>.
I basically created a FlexUnit-like testing framework for FlexJS (which could easily be updated for Royale, with a few tweaks). This is the actual repository with a proper readme that describes the project:
https://github.com/BowlerHatLLC/nextgenas-test
Some additional docs about available Metadata:
https://github.com/BowlerHatLLC/nextgenas-test/wiki/Metadata
- Josh
On 2017-11-03 09:26, Alex Harui <ah...@adobe.com.INVALID> wrote:
> I was unable to tell what it is Josh has from that link.
>
> Right now, we have 3 test subsystems:
> 1) Java-based tests using Selenium that Chris Dutz wrote
> 2) FlexUnit tests (written in AS3) that only work in Flash (in the Basic
> project)
> 3) Mustella tests that are written in MXML that run on both Flash and in
> the browser using Selenium.
>
> This thread is titled "Unit Tests" Do we have agreement on what the size
> of the unit is?
>
> Is there any advantage to trying to get FlexUnit to work without Flash?
> For example, would folks migrating have old FlexUnit tests they want to
> run?
>
> I agree with Piotr that it would be better if tests didn't have to be
> written in Java. Using a declarative language like MXML makes you more
> output-language agnostic. Mustella works by having each Mustella command
> implemented in Java for Selenium. It constrains what your test can do to
> things we know we can do on all platforms, and is agnostic about when the
> next command run, which is helpful when there are differences in the way
> the runtimes execute code and make results available. I'd have to dig
> into it again, but I think there are some issues in failure reporting in
> Selenium of Selenium thinks there is only one test step which is Java code
> that shoved a pile of JS into the browser to be executed.
>
> It would be nice if we can leverage what we have instead of building a
> whole new test system, but maybe Josh has something that is closer to what
> we are looking for.
>
> My 2 cents,
> -Alex
>
> On 11/3/17, 8:37 AM, "Josh Tynjala" <jo...@apache.org> wrote:
>
> >Please feel free to use my test runner. I'm happy to commit it somewhere
> >to make it official Apache code, if necessary.
> >
> >- Josh
> >
> >On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
> >> One topic which keeps coming up is better test coverage for Royale.
> >>
> >> I think this is becoming a critical issue for a few reasons:
> >> 1. As we get close to version 1.0 itââ¬â¢s necessary to have good test
> >>coverage for confidence of quality and confidence that we donââ¬â¢t
> >>introduce recessive bugs.
> >> 2. Itââ¬â¢s really hard to accept Github pull requests without examining
> >>the code VERY well that it does not introduce recessive bugs. CI which
> >>runs automated tests could give a preliminary test on pull requests to
> >>ensure that they donââ¬â¢t break anything. If the pull requests do break
> >>something, it allows the requester to fix the problem with confidence
> >>without taking othersââ¬â¢ time.
> >>
> >> I think we need to break up testing into pieces and figure out a
> >>strategy to implement automated tests in a way that they are
> >>maintainable.
> >>
> >> Some points:
> >> 1. I think integration into something like Travis would be very helpful.
> >> 2. I think thereââ¬â¢s a Jenkins plugin for building pull requests. Not
> >>sure how useful it is.[1]
> >> 3. Josh has created a Node.js-compatible test-runner architecture which
> >>could be useful for unit tests on parts of the framework which donââ¬â¢t
> >>rely on browser features. (i.e. models and the like) He mentioned the
> >>possibility of donating it, and I think that would be a very useful
> >>feature.
> >> 4. For UI and integration tests, there seem to be some pretty cool
> >>integrations using Selenium.[2][3]
> >> 5. I think the main testing effort needs to be using JS and something
> >>like Joshââ¬â¢s testing framework for non-UI pieces and some easy-to-use
> >>Selenium integration for visual pieces and integration tests.
> >> 6. We probably also want some API endpoints we can test off of for
> >>integration testing.
> >>
> >> Iââ¬â¢m willing to invest time into this.
> >>
> >> Itââ¬â¢s going to be a lot of work building out the tests and I think we
> >>need a plan for that. My thoughts:
> >>
> >> 1. Step one is to make it easy to write meaningful automated tests and
> >>establish a clear pattern.
> >> 2. Step two is to start writing tests starting from the
> >>most-used/easiest to beak pieces and work out from there.
> >> 3. Once the pattern is established, any new pieces MUST have testing
> >>coverage.
> >> 4. When fixing bugs, attention should be paid to adding testing for
> >>that component.
> >> 5. When a pull request comes in on a piece which does not have unit
> >>test, a test must be written before accepting the pull request. The test
> >>does not need to be written by the requester. Before examining the
> >>request, the test should be written to pass for expected behavior and
> >>fail for the bug that the pull request is attempting to fix (assuming
> >>the pull request is to fix a bug).
> >>
> >> Thoughts?
> >> Harbs
> >>
> >> P.S. Iââ¬â¢m thinking of coming to the US in late December/early January.
> >>I would be interested in getting together for a hacking session with
> >>folks who are available.
> >>
> >>
> >>[1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.
> >>jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplug
> >>in&data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b3443879
> >>4aed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2F
> >>eDEGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0
> >><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.je
> >>nkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplugin
> >>&data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794a
> >>ed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2FeD
> >>EGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0>
> >>
> >>[2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.
> >>travis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d
> >>681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%
> >>7C636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%
> >>3D&reserved=0
> >><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.tr
> >>avis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d68
> >>1ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C
> >>636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%3D
> >>&reserved=0>
> >>
> >>[3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsauce
> >>labs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f
> >>08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636453202314829
> >>220&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0
> >><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsaucela
> >>bs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f08
> >>d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C63645320231482922
> >>0&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0>
>
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
Reading the FlexUnit docs, I see that there’s a lot of advanced features (that I never even knew about).
I’m going to start with just the metadata tags that Josh wrote support for (i.e. Before, Text and After).
Maybe I’ll add Ignore, BeforeClass and AfterClass as well.
A question I have for those who have used FlexUnit is how extensively these advanced features are used?
I have to figure out the best way to run tests on both node and Flash. It might be simplest to have two separate runner classes…
Harbs
> On Nov 4, 2017, at 7:20 PM, Harbs <ha...@gmail.com> wrote:
>
> Wow. I don’t think I ever noticed the tutorial for FlexUnit on the Flex site:
> https://flex.apache.org/flexunit/tutorial/ <https://flex.apache.org/flexunit/tutorial/>
>
> Who put that together? Looks like a lot of work went into that.
>
> Harbs
>
>> On Nov 4, 2017, at 6:59 PM, Harbs <harbs.lists@gmail.com <ma...@gmail.com>> wrote:
>>
>> By “Unit Tests”, I really meant tests which are UI agnostic. I’m not concerned by size of “Unit” tests, I’m more concerned about feature coverage. When I think of unit tests in this context I mean FlexUnit-type tests. The problem with FlexUnit, is that it only tests in Flash. Josh’s work solves that problem. Ideally, we’d have tests which are run on both platforms. I think We can use FlexUnit on Flash and Josh’s implementation to run the exact same tests in node.
>>
>> To me, it’s easier to tackle this problem if we first get “unit” tests working that are platform independent.
>>
>> Once that’s working, we can figure out how much the Unit-type tests can be extended to test UI features and/or Mustella can be used. Mustella is a big mystery to me.
>>
>> Whatever the strategy, the tests need to be easy to write. Maybe it already is easy to write mustella tests, but we need more documentation. I don’t know.
>>
>> Either way, I will try to add Josh’s code and write some tests and see how it goes.
>>
>> Harbs
>>
>>> On Nov 3, 2017, at 6:26 PM, Alex Harui <aharui@adobe.com.INVALID <ma...@adobe.com.INVALID>> wrote:
>>>
>>> I was unable to tell what it is Josh has from that link.
>>>
>>> Right now, we have 3 test subsystems:
>>> 1) Java-based tests using Selenium that Chris Dutz wrote
>>> 2) FlexUnit tests (written in AS3) that only work in Flash (in the Basic
>>> project)
>>> 3) Mustella tests that are written in MXML that run on both Flash and in
>>> the browser using Selenium.
>>>
>>> This thread is titled "Unit Tests" Do we have agreement on what the size
>>> of the unit is?
>>>
>>> Is there any advantage to trying to get FlexUnit to work without Flash?
>>> For example, would folks migrating have old FlexUnit tests they want to
>>> run?
>>>
>>> I agree with Piotr that it would be better if tests didn't have to be
>>> written in Java. Using a declarative language like MXML makes you more
>>> output-language agnostic. Mustella works by having each Mustella command
>>> implemented in Java for Selenium. It constrains what your test can do to
>>> things we know we can do on all platforms, and is agnostic about when the
>>> next command run, which is helpful when there are differences in the way
>>> the runtimes execute code and make results available. I'd have to dig
>>> into it again, but I think there are some issues in failure reporting in
>>> Selenium of Selenium thinks there is only one test step which is Java code
>>> that shoved a pile of JS into the browser to be executed.
>>>
>>> It would be nice if we can leverage what we have instead of building a
>>> whole new test system, but maybe Josh has something that is closer to what
>>> we are looking for.
>>>
>>> My 2 cents,
>>> -Alex
>>>
>>> On 11/3/17, 8:37 AM, "Josh Tynjala" <joshtynjala@apache.org <ma...@apache.org>> wrote:
>>>
>>>> Please feel free to use my test runner. I'm happy to commit it somewhere
>>>> to make it official Apache code, if necessary.
>>>>
>>>> - Josh
>>>>
>>>> On 2017-11-03 04:19, Harbs <harbs.lists@gmail.com <ma...@gmail.com>> wrote:
>>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>>
>>>>> I think this is becoming a critical issue for a few reasons:
>>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>>>> coverage for confidence of quality and confidence that we don’t
>>>>> introduce recessive bugs.
>>>>> 2. It’s really hard to accept Github pull requests without examining
>>>>> the code VERY well that it does not introduce recessive bugs. CI which
>>>>> runs automated tests could give a preliminary test on pull requests to
>>>>> ensure that they don’t break anything. If the pull requests do break
>>>>> something, it allows the requester to fix the problem with confidence
>>>>> without taking others’ time.
>>>>>
>>>>> I think we need to break up testing into pieces and figure out a
>>>>> strategy to implement automated tests in a way that they are
>>>>> maintainable.
>>>>>
>>>>> Some points:
>>>>> 1. I think integration into something like Travis would be very helpful.
>>>>> 2. I think there’s a Jenkins plugin for building pull requests. Not
>>>>> sure how useful it is.[1]
>>>>> 3. Josh has created a Node.js-compatible test-runner architecture which
>>>>> could be useful for unit tests on parts of the framework which don’t
>>>>> rely on browser features. (i.e. models and the like) He mentioned the
>>>>> possibility of donating it, and I think that would be a very useful
>>>>> feature.
>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>>> integrations using Selenium.[2][3]
>>>>> 5. I think the main testing effort needs to be using JS and something
>>>>> like Josh’s testing framework for non-UI pieces and some easy-to-use
>>>>> Selenium integration for visual pieces and integration tests.
>>>>> 6. We probably also want some API endpoints we can test off of for
>>>>> integration testing.
>>>>>
>>>>> I’m willing to invest time into this.
>>>>>
>>>>> It’s going to be a lot of work building out the tests and I think we
>>>>> need a plan for that. My thoughts:
>>>>>
>>>>> 1. Step one is to make it easy to write meaningful automated tests and
>>>>> establish a clear pattern.
>>>>> 2. Step two is to start writing tests starting from the
>>>>> most-used/easiest to beak pieces and work out from there.
>>>>> 3. Once the pattern is established, any new pieces MUST have testing
>>>>> coverage.
>>>>> 4. When fixing bugs, attention should be paid to adding testing for
>>>>> that component.
>>>>> 5. When a pull request comes in on a piece which does not have unit
>>>>> test, a test must be written before accepting the pull request. The test
>>>>> does not need to be written by the requester. Before examining the
>>>>> request, the test should be written to pass for expected behavior and
>>>>> fail for the bug that the pull request is attempting to fix (assuming
>>>>> the pull request is to fix a bug).
>>>>>
>>>>> Thoughts?
>>>>> Harbs
>>>>>
>>>>> P.S. I’m thinking of coming to the US in late December/early January.
>>>>> I would be interested in getting together for a hacking session with
>>>>> folks who are available.
>>>>>
>>>>>
>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki>.
>>>>> jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplug
>>>>> in&data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b3443879
>>>>> 4aed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2F
>>>>> eDEGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0
>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.je <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.je>
>>>>> nkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplugin
>>>>> &data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794a
>>>>> ed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2FeD
>>>>> EGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0>
>>>>>
>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs>.
>>>>> travis-ci.com <http://travis-ci.com/>%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d
>>>>> 681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%
>>>>> 7C636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%
>>>>> 3D&reserved=0
>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.tr <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.tr>
>>>>> avis-ci.com <http://avis-ci.com/>%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d68
>>>>> 1ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C
>>>>> 636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%3D
>>>>> &reserved=0>
>>>>>
>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsauce <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsauce>
>>>>> labs.com <http://labs.com/>%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f
>>>>> 08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636453202314829
>>>>> 220&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0
>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsaucela <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsaucela>
>>>>> bs.com <http://bs.com/>%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f08
>>>>> d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C63645320231482922
>>>>> 0&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0>
>>>
>>
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
Wow. I don’t think I ever noticed the tutorial for FlexUnit on the Flex site:
https://flex.apache.org/flexunit/tutorial/ <https://flex.apache.org/flexunit/tutorial/>
Who put that together? Looks like a lot of work went into that.
Harbs
> On Nov 4, 2017, at 6:59 PM, Harbs <ha...@gmail.com> wrote:
>
> By “Unit Tests”, I really meant tests which are UI agnostic. I’m not concerned by size of “Unit” tests, I’m more concerned about feature coverage. When I think of unit tests in this context I mean FlexUnit-type tests. The problem with FlexUnit, is that it only tests in Flash. Josh’s work solves that problem. Ideally, we’d have tests which are run on both platforms. I think We can use FlexUnit on Flash and Josh’s implementation to run the exact same tests in node.
>
> To me, it’s easier to tackle this problem if we first get “unit” tests working that are platform independent.
>
> Once that’s working, we can figure out how much the Unit-type tests can be extended to test UI features and/or Mustella can be used. Mustella is a big mystery to me.
>
> Whatever the strategy, the tests need to be easy to write. Maybe it already is easy to write mustella tests, but we need more documentation. I don’t know.
>
> Either way, I will try to add Josh’s code and write some tests and see how it goes.
>
> Harbs
>
>> On Nov 3, 2017, at 6:26 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>>
>> I was unable to tell what it is Josh has from that link.
>>
>> Right now, we have 3 test subsystems:
>> 1) Java-based tests using Selenium that Chris Dutz wrote
>> 2) FlexUnit tests (written in AS3) that only work in Flash (in the Basic
>> project)
>> 3) Mustella tests that are written in MXML that run on both Flash and in
>> the browser using Selenium.
>>
>> This thread is titled "Unit Tests" Do we have agreement on what the size
>> of the unit is?
>>
>> Is there any advantage to trying to get FlexUnit to work without Flash?
>> For example, would folks migrating have old FlexUnit tests they want to
>> run?
>>
>> I agree with Piotr that it would be better if tests didn't have to be
>> written in Java. Using a declarative language like MXML makes you more
>> output-language agnostic. Mustella works by having each Mustella command
>> implemented in Java for Selenium. It constrains what your test can do to
>> things we know we can do on all platforms, and is agnostic about when the
>> next command run, which is helpful when there are differences in the way
>> the runtimes execute code and make results available. I'd have to dig
>> into it again, but I think there are some issues in failure reporting in
>> Selenium of Selenium thinks there is only one test step which is Java code
>> that shoved a pile of JS into the browser to be executed.
>>
>> It would be nice if we can leverage what we have instead of building a
>> whole new test system, but maybe Josh has something that is closer to what
>> we are looking for.
>>
>> My 2 cents,
>> -Alex
>>
>> On 11/3/17, 8:37 AM, "Josh Tynjala" <jo...@apache.org> wrote:
>>
>>> Please feel free to use my test runner. I'm happy to commit it somewhere
>>> to make it official Apache code, if necessary.
>>>
>>> - Josh
>>>
>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>
>>>> I think this is becoming a critical issue for a few reasons:
>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>>> coverage for confidence of quality and confidence that we don’t
>>>> introduce recessive bugs.
>>>> 2. It’s really hard to accept Github pull requests without examining
>>>> the code VERY well that it does not introduce recessive bugs. CI which
>>>> runs automated tests could give a preliminary test on pull requests to
>>>> ensure that they don’t break anything. If the pull requests do break
>>>> something, it allows the requester to fix the problem with confidence
>>>> without taking others’ time.
>>>>
>>>> I think we need to break up testing into pieces and figure out a
>>>> strategy to implement automated tests in a way that they are
>>>> maintainable.
>>>>
>>>> Some points:
>>>> 1. I think integration into something like Travis would be very helpful.
>>>> 2. I think there’s a Jenkins plugin for building pull requests. Not
>>>> sure how useful it is.[1]
>>>> 3. Josh has created a Node.js-compatible test-runner architecture which
>>>> could be useful for unit tests on parts of the framework which don’t
>>>> rely on browser features. (i.e. models and the like) He mentioned the
>>>> possibility of donating it, and I think that would be a very useful
>>>> feature.
>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>> integrations using Selenium.[2][3]
>>>> 5. I think the main testing effort needs to be using JS and something
>>>> like Josh’s testing framework for non-UI pieces and some easy-to-use
>>>> Selenium integration for visual pieces and integration tests.
>>>> 6. We probably also want some API endpoints we can test off of for
>>>> integration testing.
>>>>
>>>> I’m willing to invest time into this.
>>>>
>>>> It’s going to be a lot of work building out the tests and I think we
>>>> need a plan for that. My thoughts:
>>>>
>>>> 1. Step one is to make it easy to write meaningful automated tests and
>>>> establish a clear pattern.
>>>> 2. Step two is to start writing tests starting from the
>>>> most-used/easiest to beak pieces and work out from there.
>>>> 3. Once the pattern is established, any new pieces MUST have testing
>>>> coverage.
>>>> 4. When fixing bugs, attention should be paid to adding testing for
>>>> that component.
>>>> 5. When a pull request comes in on a piece which does not have unit
>>>> test, a test must be written before accepting the pull request. The test
>>>> does not need to be written by the requester. Before examining the
>>>> request, the test should be written to pass for expected behavior and
>>>> fail for the bug that the pull request is attempting to fix (assuming
>>>> the pull request is to fix a bug).
>>>>
>>>> Thoughts?
>>>> Harbs
>>>>
>>>> P.S. I’m thinking of coming to the US in late December/early January.
>>>> I would be interested in getting together for a hacking session with
>>>> folks who are available.
>>>>
>>>>
>>>> [1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.
>>>> jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplug
>>>> in&data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b3443879
>>>> 4aed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2F
>>>> eDEGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0
>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.je
>>>> nkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplugin
>>>> &data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794a
>>>> ed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2FeD
>>>> EGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0>
>>>>
>>>> [2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.
>>>> travis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d
>>>> 681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%
>>>> 7C636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%
>>>> 3D&reserved=0
>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.tr
>>>> avis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d68
>>>> 1ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C
>>>> 636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%3D
>>>> &reserved=0>
>>>>
>>>> [3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsauce
>>>> labs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f
>>>> 08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636453202314829
>>>> 220&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0
>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsaucela
>>>> bs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f08
>>>> d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C63645320231482922
>>>> 0&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0>
>>
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
By “Unit Tests”, I really meant tests which are UI agnostic. I’m not concerned by size of “Unit” tests, I’m more concerned about feature coverage. When I think of unit tests in this context I mean FlexUnit-type tests. The problem with FlexUnit, is that it only tests in Flash. Josh’s work solves that problem. Ideally, we’d have tests which are run on both platforms. I think We can use FlexUnit on Flash and Josh’s implementation to run the exact same tests in node.
To me, it’s easier to tackle this problem if we first get “unit” tests working that are platform independent.
Once that’s working, we can figure out how much the Unit-type tests can be extended to test UI features and/or Mustella can be used. Mustella is a big mystery to me.
Whatever the strategy, the tests need to be easy to write. Maybe it already is easy to write mustella tests, but we need more documentation. I don’t know.
Either way, I will try to add Josh’s code and write some tests and see how it goes.
Harbs
> On Nov 3, 2017, at 6:26 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>
> I was unable to tell what it is Josh has from that link.
>
> Right now, we have 3 test subsystems:
> 1) Java-based tests using Selenium that Chris Dutz wrote
> 2) FlexUnit tests (written in AS3) that only work in Flash (in the Basic
> project)
> 3) Mustella tests that are written in MXML that run on both Flash and in
> the browser using Selenium.
>
> This thread is titled "Unit Tests" Do we have agreement on what the size
> of the unit is?
>
> Is there any advantage to trying to get FlexUnit to work without Flash?
> For example, would folks migrating have old FlexUnit tests they want to
> run?
>
> I agree with Piotr that it would be better if tests didn't have to be
> written in Java. Using a declarative language like MXML makes you more
> output-language agnostic. Mustella works by having each Mustella command
> implemented in Java for Selenium. It constrains what your test can do to
> things we know we can do on all platforms, and is agnostic about when the
> next command run, which is helpful when there are differences in the way
> the runtimes execute code and make results available. I'd have to dig
> into it again, but I think there are some issues in failure reporting in
> Selenium of Selenium thinks there is only one test step which is Java code
> that shoved a pile of JS into the browser to be executed.
>
> It would be nice if we can leverage what we have instead of building a
> whole new test system, but maybe Josh has something that is closer to what
> we are looking for.
>
> My 2 cents,
> -Alex
>
> On 11/3/17, 8:37 AM, "Josh Tynjala" <jo...@apache.org> wrote:
>
>> Please feel free to use my test runner. I'm happy to commit it somewhere
>> to make it official Apache code, if necessary.
>>
>> - Josh
>>
>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>>> One topic which keeps coming up is better test coverage for Royale.
>>>
>>> I think this is becoming a critical issue for a few reasons:
>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>> coverage for confidence of quality and confidence that we don’t
>>> introduce recessive bugs.
>>> 2. It’s really hard to accept Github pull requests without examining
>>> the code VERY well that it does not introduce recessive bugs. CI which
>>> runs automated tests could give a preliminary test on pull requests to
>>> ensure that they don’t break anything. If the pull requests do break
>>> something, it allows the requester to fix the problem with confidence
>>> without taking others’ time.
>>>
>>> I think we need to break up testing into pieces and figure out a
>>> strategy to implement automated tests in a way that they are
>>> maintainable.
>>>
>>> Some points:
>>> 1. I think integration into something like Travis would be very helpful.
>>> 2. I think there’s a Jenkins plugin for building pull requests. Not
>>> sure how useful it is.[1]
>>> 3. Josh has created a Node.js-compatible test-runner architecture which
>>> could be useful for unit tests on parts of the framework which don’t
>>> rely on browser features. (i.e. models and the like) He mentioned the
>>> possibility of donating it, and I think that would be a very useful
>>> feature.
>>> 4. For UI and integration tests, there seem to be some pretty cool
>>> integrations using Selenium.[2][3]
>>> 5. I think the main testing effort needs to be using JS and something
>>> like Josh’s testing framework for non-UI pieces and some easy-to-use
>>> Selenium integration for visual pieces and integration tests.
>>> 6. We probably also want some API endpoints we can test off of for
>>> integration testing.
>>>
>>> I’m willing to invest time into this.
>>>
>>> It’s going to be a lot of work building out the tests and I think we
>>> need a plan for that. My thoughts:
>>>
>>> 1. Step one is to make it easy to write meaningful automated tests and
>>> establish a clear pattern.
>>> 2. Step two is to start writing tests starting from the
>>> most-used/easiest to beak pieces and work out from there.
>>> 3. Once the pattern is established, any new pieces MUST have testing
>>> coverage.
>>> 4. When fixing bugs, attention should be paid to adding testing for
>>> that component.
>>> 5. When a pull request comes in on a piece which does not have unit
>>> test, a test must be written before accepting the pull request. The test
>>> does not need to be written by the requester. Before examining the
>>> request, the test should be written to pass for expected behavior and
>>> fail for the bug that the pull request is attempting to fix (assuming
>>> the pull request is to fix a bug).
>>>
>>> Thoughts?
>>> Harbs
>>>
>>> P.S. I’m thinking of coming to the US in late December/early January.
>>> I would be interested in getting together for a hacking session with
>>> folks who are available.
>>>
>>>
>>> [1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.
>>> jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplug
>>> in&data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b3443879
>>> 4aed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2F
>>> eDEGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0
>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.je
>>> nkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplugin
>>> &data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794a
>>> ed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2FeD
>>> EGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0>
>>>
>>> [2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.
>>> travis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d
>>> 681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%
>>> 7C636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%
>>> 3D&reserved=0
>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.tr
>>> avis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d68
>>> 1ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C
>>> 636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%3D
>>> &reserved=0>
>>>
>>> [3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsauce
>>> labs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f
>>> 08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636453202314829
>>> 220&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0
>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsaucela
>>> bs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f08
>>> d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C63645320231482922
>>> 0&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0>
>
Re: Unit Tests et. al.
Posted by Alex Harui <ah...@adobe.com.INVALID>.
I was unable to tell what it is Josh has from that link.
Right now, we have 3 test subsystems:
1) Java-based tests using Selenium that Chris Dutz wrote
2) FlexUnit tests (written in AS3) that only work in Flash (in the Basic
project)
3) Mustella tests that are written in MXML that run on both Flash and in
the browser using Selenium.
This thread is titled "Unit Tests" Do we have agreement on what the size
of the unit is?
Is there any advantage to trying to get FlexUnit to work without Flash?
For example, would folks migrating have old FlexUnit tests they want to
run?
I agree with Piotr that it would be better if tests didn't have to be
written in Java. Using a declarative language like MXML makes you more
output-language agnostic. Mustella works by having each Mustella command
implemented in Java for Selenium. It constrains what your test can do to
things we know we can do on all platforms, and is agnostic about when the
next command run, which is helpful when there are differences in the way
the runtimes execute code and make results available. I'd have to dig
into it again, but I think there are some issues in failure reporting in
Selenium of Selenium thinks there is only one test step which is Java code
that shoved a pile of JS into the browser to be executed.
It would be nice if we can leverage what we have instead of building a
whole new test system, but maybe Josh has something that is closer to what
we are looking for.
My 2 cents,
-Alex
On 11/3/17, 8:37 AM, "Josh Tynjala" <jo...@apache.org> wrote:
>Please feel free to use my test runner. I'm happy to commit it somewhere
>to make it official Apache code, if necessary.
>
>- Josh
>
>On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>> One topic which keeps coming up is better test coverage for Royale.
>>
>> I think this is becoming a critical issue for a few reasons:
>> 1. As we get close to version 1.0 it’s necessary to have good test
>>coverage for confidence of quality and confidence that we don’t
>>introduce recessive bugs.
>> 2. It’s really hard to accept Github pull requests without examining
>>the code VERY well that it does not introduce recessive bugs. CI which
>>runs automated tests could give a preliminary test on pull requests to
>>ensure that they don’t break anything. If the pull requests do break
>>something, it allows the requester to fix the problem with confidence
>>without taking others’ time.
>>
>> I think we need to break up testing into pieces and figure out a
>>strategy to implement automated tests in a way that they are
>>maintainable.
>>
>> Some points:
>> 1. I think integration into something like Travis would be very helpful.
>> 2. I think there’s a Jenkins plugin for building pull requests. Not
>>sure how useful it is.[1]
>> 3. Josh has created a Node.js-compatible test-runner architecture which
>>could be useful for unit tests on parts of the framework which don’t
>>rely on browser features. (i.e. models and the like) He mentioned the
>>possibility of donating it, and I think that would be a very useful
>>feature.
>> 4. For UI and integration tests, there seem to be some pretty cool
>>integrations using Selenium.[2][3]
>> 5. I think the main testing effort needs to be using JS and something
>>like Josh’s testing framework for non-UI pieces and some easy-to-use
>>Selenium integration for visual pieces and integration tests.
>> 6. We probably also want some API endpoints we can test off of for
>>integration testing.
>>
>> I’m willing to invest time into this.
>>
>> It’s going to be a lot of work building out the tests and I think we
>>need a plan for that. My thoughts:
>>
>> 1. Step one is to make it easy to write meaningful automated tests and
>>establish a clear pattern.
>> 2. Step two is to start writing tests starting from the
>>most-used/easiest to beak pieces and work out from there.
>> 3. Once the pattern is established, any new pieces MUST have testing
>>coverage.
>> 4. When fixing bugs, attention should be paid to adding testing for
>>that component.
>> 5. When a pull request comes in on a piece which does not have unit
>>test, a test must be written before accepting the pull request. The test
>>does not need to be written by the requester. Before examining the
>>request, the test should be written to pass for expected behavior and
>>fail for the bug that the pull request is attempting to fix (assuming
>>the pull request is to fix a bug).
>>
>> Thoughts?
>> Harbs
>>
>> P.S. I’m thinking of coming to the US in late December/early January.
>>I would be interested in getting together for a hacking session with
>>folks who are available.
>>
>>
>>[1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.
>>jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplug
>>in&data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b3443879
>>4aed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2F
>>eDEGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0
>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.je
>>nkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull%2Brequest%2Bbuilder%2Bplugin
>>&data=02%7C01%7C%7C36d681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794a
>>ed2c178decee1%7C0%7C0%7C636453202314829220&sdata=dqT0Nao8HHzlkWmk6zy%2FeD
>>EGi6nXYkM8P0%2Bbibe4NA4%3D&reserved=0>
>>
>>[2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.
>>travis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d
>>681ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%
>>7C636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%
>>3D&reserved=0
>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.tr
>>avis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C36d68
>>1ea88ba4315312f08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C
>>636453202314829220&sdata=9D9riTJ29rqNonT81v%2FRtdpC6J8titW7LIUb07LLXa4%3D
>>&reserved=0>
>>
>>[3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsauce
>>labs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f
>>08d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636453202314829
>>220&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0
>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsaucela
>>bs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C36d681ea88ba4315312f08
>>d522d0bf50%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C63645320231482922
>>0&sdata=WNzEofopPA6YaHS51O%2BCx6Zrsc2qLOFcSWMUUXssD3M%3D&reserved=0>
Re: Unit Tests et. al.
Posted by Josh Tynjala <jo...@apache.org>.
Sorry for the late response. Yes, go ahead!
- Josh
On 2017-11-04 11:30, Harbs <ha...@gmail.com> wrote:
> Awesome!
>
> I assume that this means I can code the code to the Royale repo under Apache 2. Right?
>
> Harbs
>
> > On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org> wrote:
> >
> > Please feel free to use my test runner. I'm happy to commit it somewhere to make it official Apache code, if necessary.
> >
> > - Josh
> >
> > On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
> >> One topic which keeps coming up is better test coverage for Royale.
> >>
> >> I think this is becoming a critical issue for a few reasons:
> >> 1. As we get close to version 1.0 itâÂÂs necessary to have good test coverage for confidence of quality and confidence that we donâÂÂt introduce recessive bugs.
> >> 2. ItâÂÂs really hard to accept Github pull requests without examining the code VERY well that it does not introduce recessive bugs. CI which runs automated tests could give a preliminary test on pull requests to ensure that they donâÂÂt break anything. If the pull requests do break something, it allows the requester to fix the problem with confidence without taking othersâ time.
> >>
> >> I think we need to break up testing into pieces and figure out a strategy to implement automated tests in a way that they are maintainable.
> >>
> >> Some points:
> >> 1. I think integration into something like Travis would be very helpful.
> >> 2. I think thereâÂÂs a Jenkins plugin for building pull requests. Not sure how useful it is.[1]
> >> 3. Josh has created a Node.js-compatible test-runner architecture which could be useful for unit tests on parts of the framework which donâÂÂt rely on browser features. (i.e. models and the like) He mentioned the possibility of donating it, and I think that would be a very useful feature.
> >> 4. For UI and integration tests, there seem to be some pretty cool integrations using Selenium.[2][3]
> >> 5. I think the main testing effort needs to be using JS and something like JoshâÂÂs testing framework for non-UI pieces and some easy-to-use Selenium integration for visual pieces and integration tests.
> >> 6. We probably also want some API endpoints we can test off of for integration testing.
> >>
> >> IâÂÂm willing to invest time into this.
> >>
> >> ItâÂÂs going to be a lot of work building out the tests and I think we need a plan for that. My thoughts:
> >>
> >> 1. Step one is to make it easy to write meaningful automated tests and establish a clear pattern.
> >> 2. Step two is to start writing tests starting from the most-used/easiest to beak pieces and work out from there.
> >> 3. Once the pattern is established, any new pieces MUST have testing coverage.
> >> 4. When fixing bugs, attention should be paid to adding testing for that component.
> >> 5. When a pull request comes in on a piece which does not have unit test, a test must be written before accepting the pull request. The test does not need to be written by the requester. Before examining the request, the test should be written to pass for expected behavior and fail for the bug that the pull request is attempting to fix (assuming the pull request is to fix a bug).
> >>
> >> Thoughts?
> >> Harbs
> >>
> >> P.S. IâÂÂm thinking of coming to the US in late December/early January. I would be interested in getting together for a hacking session with folks who are available.
> >>
> >> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin <https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin>
> >> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>
> >> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/products/integrations>
>
>
Re: Unit Tests et. al.
Posted by Greg Dove <gr...@gmail.com>.
Great to hear. I did try to communicate about that at the time, but
probably did not do a great job. Hope you find something useful there to
combine with Josh's work.
fyi I hope to get back to some work on things before too long. I do have
some WIP on a compiler optimization I want to continue, but it may still be
a couple of months because of basic income-related priorities.
On Sun, Nov 5, 2017 at 9:58 AM, Harbs <ha...@gmail.com> wrote:
> FYI, It compiled and worked on the first try. :-)
>
> The test view is very cool. I’m not sure if I missed it when you mentioned
> this, but I like. :-)
>
> Harbs
>
> > On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
> >
> > Harbs, I don't know if it still works since changes to Royale, but I had
> > something rudimentary for cross-target unit testing quite shortly after
> > working on the reflection support last year, in fact that was my primary
> > reason for choosing to focus on reflection at the time.
>
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
FYI, It compiled and worked on the first try. :-)
The test view is very cool. I’m not sure if I missed it when you mentioned this, but I like. :-)
Harbs
> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
>
> Harbs, I don't know if it still works since changes to Royale, but I had
> something rudimentary for cross-target unit testing quite shortly after
> working on the reflection support last year, in fact that was my primary
> reason for choosing to focus on reflection at the time.
Re: Unit Tests et. al.
Posted by Piotr Zarzycki <pi...@gmail.com>.
Hi Harbs,
Can you raise for me github issue for Maven, since you have merged and we
are going to proceed on develop I prefer have it.
Piotr
2017-11-06 10:42 GMT+01:00 Harbs <ha...@gmail.com>:
> Well, Josh reversed the order of the arguments, and I think that makes
> sense. The string is optional and it was at the beginning. It’s now the
> optional last argument and all the arguments are as typed as they can be.
> (compare arguments are of type Object.)
>
> I just merged the testing branch into develop, so tests can be added on
> the develop branch. The js testing is not functional yet, but the tests can
> be written in a platform agnostic way.
>
> More to come…
>
> Harbs
>
> > On Nov 6, 2017, at 12:51 AM, Piotr Zarzycki <pi...@gmail.com>
> wrote:
> >
> > My guess is that, cause this approach emulates JUnit implementation.
> Let's
> > say assertFalse - you can pass there one argument Boolean or two
> arguments
> > Boolean and message. In AS3 there is no ability to create two functions
> > with the same name. But in Java method overloading probably exists, so in
> > JUnit there is probably:
> >
> > assertFalse(Boolean param);
> > assertFalse(Boolean param, String);
> >
> > Summarize everything - I see only some AS3 language restriction as an
> > reason.
> >
> > Piotr
> >
> >
> >
> > 2017-11-05 23:14 GMT+01:00 Harbs <ha...@gmail.com>:
> >
> >> BTW:
> >>
> >> Was there a good reason FlexUnit uses the spread operator for assert
> >> arguments instead of typed arguments?
> >>
> >>> On Nov 5, 2017, at 11:36 PM, Harbs <ha...@gmail.com> wrote:
> >>>
> >>> Some progress on the unit tests:
> >>>
> >>> I got the Core tests to run on the swf side when doing a full build. In
> >> order to do that I needed to move the running of the tests until after
> >> Basic finished building so the test app could be built.
> >>>
> >>> We’re going to need to do something similar with the Maven build.
> >>>
> >>> The Testing project is compiling and is usable (on a basic level) too.
> >> Tests should run with both a SWF and JS target. The SWF target uses
> >> FlexUnit, while the JS target uses Josh’s implementation.
> >>>
> >>> I also migrated the Core tests to use the new project and that is
> >> building as well.
> >>>
> >>> I still need to add the bits that Greg wrote and wire up some JS tests.
> >>>
> >>> Hopefully I’ll make some more progress tomorrow.
> >>>
> >>> Harbs
> >>>
> >>>> On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID>
> >> wrote:
> >>>>
> >>>> Those warnings do not seem to affect the success of the tests on SWF
> for
> >>>> me.
> >>>>
> >>>> IMO, the big challenge is in capturing test results so it can be
> >> reported
> >>>> in a CI dashboard. Having a GUI display results in Flash or in a
> >> browser
> >>>> is a great first step, but I believe going from there to getting
> >> Selenium
> >>>> to report individual test failures may be trickier.
> >>>>
> >>>> Peter tried to write up Mustella a long time ago. It might need
> >> updating.
> >>>> See:
> >>>> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
> >>>>
> >>>> The advantage of declarative testing languages is that you get to
> >>>> constrain the kinds of things that a test can do, and abstract when
> the
> >>>> test harness does it. It is much harder to interrupt sequential lines
> >> of
> >>>> ActionScript/JavaScript.
> >>>>
> >>>> HTH,
> >>>> -Alex
> >>>>
> >>>> On 11/4/17, 3:13 PM, "Harbs" <ha...@gmail.com> wrote:
> >>>>
> >>>>> It looks like FlexUnit is currently broken.
> >>>>>
> >>>>> I’m getting the following output when trying to run the FlexUnit
> tests
> >> on
> >>>>> Core and Basic:
> >>>>>
> >>>>> [mxmlc]
> >>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> >> FlexUnit4/target/flexuni
> >>>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
> >>>>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description
> in
> >> the
> >>>>> SWC
> >>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> >> FlexUnit4/target/flexuni
> >>>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
> >>>>> [mxmlc]
> >>>>> [mxmlc]
> >>>>> [mxmlc]
> >>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> >> FlexUnit4/target/flexuni
> >>>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
> >>>>> mx.utils.StringUtil depended on by flexunit.framework.
> AsyncTestHelper
> >> in
> >>>>> the SWC
> >>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> >> FlexUnit4/target/flexuni
> >>>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
> >>>>>
> >>>>> Has anyone looked at this yet?
> >>>>>
> >>>>> Harbs
> >>>>>
> >>>>>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
> >>>>>>
> >>>>>> Thanks! That looks very useful.
> >>>>>>
> >>>>>> I started work on a feature/testing branch. I’ll try to merge what
> you
> >>>>>> did into what Josh did. I’m going to try to get the existing Core
> >> and/or
> >>>>>> Basic tests working in both swf and node js output tomorrow. We’ll
> see
> >>>>>> how well I do… ;-)
> >>>>>>
> >>>>>> We might need different configs for different testing environments,
> >> but
> >>>>>> I’ll see if we can keep the divergence of the environs is minimal as
> >>>>>> possible.
> >>>>>>
> >>>>>> Once I get the basics worked out, I’ll likely have an idea what
> others
> >>>>>> can help with. Let’s see if we can whip the Royale tests into shape.
> >> :-)
> >>>>>>
> >>>>>> Harbs
> >>>>>>
> >>>>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>> Harbs, I don't know if it still works since changes to Royale, but
> I
> >>>>>>> had
> >>>>>>> something rudimentary for cross-target unit testing quite shortly
> >> after
> >>>>>>> working on the reflection support last year, in fact that was my
> >>>>>>> primary
> >>>>>>> reason for choosing to focus on reflection at the time.
> >>>>>>>
> >>>>>>> It was visual output only (i.e. just to look at the test results
> >>>>>>> output in
> >>>>>>> the browser) and the goal was to get some compatibility with
> flexunit
> >>>>>>> (so
> >>>>>>> that the same swf-based flexunit tests that were running in the
> build
> >>>>>>> -e..g
> >>>>>>> BinaryData tests- could be run visually side by side between swf
> and
> >>>>>>> js).
> >>>>>>> I also added a new TestVariance metadata to account for known
> >>>>>>> (expected?)
> >>>>>>> variation between swf and js. This was needed to cover (for
> example)
> >>>>>>> things
> >>>>>>> like testing Reflection classes themselves, where the results can
> be
> >>>>>>> different between the targets based on the 'native' base classes
> >> (e..g
> >>>>>>> EventDispatcher inheritance chain).
> >>>>>>>
> >>>>>>> Some of that code might also be useful to harvest for what you are
> >>>>>>> doing,
> >>>>>>> I'm not sure...
> >>>>>>>
> >>>>>>> It's here:
> >>>>>>>
> >>>>>>> https://na01.safelinks.protection.outlook.com/?url=
> >> https%3A%2F%2Fgithub.
> >>>>>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%
> >> 2Fmanualtests%2FUnitTests&da
> >>>>>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> >> 5d35%7Cfa7b1b5a7b34438794aed
> >>>>>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=
> >> DX9gXiLqUoFQizp6xrakUwxjl
> >>>>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com>
> wrote:
> >>>>>>>
> >>>>>>>> Awesome!
> >>>>>>>>
> >>>>>>>> I assume that this means I can code the code to the Royale repo
> >> under
> >>>>>>>> Apache 2. Right?
> >>>>>>>>
> >>>>>>>> Harbs
> >>>>>>>>
> >>>>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <joshtynjala@apache.org
> >
> >>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Please feel free to use my test runner. I'm happy to commit it
> >>>>>>>>> somewhere
> >>>>>>>> to make it official Apache code, if necessary.
> >>>>>>>>>
> >>>>>>>>> - Josh
> >>>>>>>>>
> >>>>>>>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
> >>>>>>>>>> One topic which keeps coming up is better test coverage for
> >> Royale.
> >>>>>>>>>>
> >>>>>>>>>> I think this is becoming a critical issue for a few reasons:
> >>>>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good
> >> test
> >>>>>>>> coverage for confidence of quality and confidence that we don’t
> >>>>>>>> introduce
> >>>>>>>> recessive bugs.
> >>>>>>>>>> 2. It’s really hard to accept Github pull requests without
> >>>>>>>>>> examining
> >>>>>>>> the code VERY well that it does not introduce recessive bugs. CI
> >>>>>>>> which runs
> >>>>>>>> automated tests could give a preliminary test on pull requests to
> >>>>>>>> ensure
> >>>>>>>> that they don’t break anything. If the pull requests do break
> >>>>>>>> something,
> >>>>>>>> it allows the requester to fix the problem with confidence without
> >>>>>>>> taking
> >>>>>>>> others’ time.
> >>>>>>>>>>
> >>>>>>>>>> I think we need to break up testing into pieces and figure out a
> >>>>>>>> strategy to implement automated tests in a way that they are
> >>>>>>>> maintainable.
> >>>>>>>>>>
> >>>>>>>>>> Some points:
> >>>>>>>>>> 1. I think integration into something like Travis would be very
> >>>>>>>>>> helpful.
> >>>>>>>>>> 2. I think there’s a Jenkins plugin for building pull
> requests.
> >>>>>>>>>> Not
> >>>>>>>> sure how useful it is.[1]
> >>>>>>>>>> 3. Josh has created a Node.js-compatible test-runner
> architecture
> >>>>>>>>>> which
> >>>>>>>> could be useful for unit tests on parts of the framework which
> >>>>>>>> don’t rely
> >>>>>>>> on browser features. (i.e. models and the like) He mentioned the
> >>>>>>>> possibility of donating it, and I think that would be a very
> useful
> >>>>>>>> feature.
> >>>>>>>>>> 4. For UI and integration tests, there seem to be some pretty
> cool
> >>>>>>>> integrations using Selenium.[2][3]
> >>>>>>>>>> 5. I think the main testing effort needs to be using JS and
> >>>>>>>>>> something
> >>>>>>>> like Josh’s testing framework for non-UI pieces and some
> >> easy-to-use
> >>>>>>>> Selenium integration for visual pieces and integration tests.
> >>>>>>>>>> 6. We probably also want some API endpoints we can test off of
> for
> >>>>>>>> integration testing.
> >>>>>>>>>>
> >>>>>>>>>> I’m willing to invest time into this.
> >>>>>>>>>>
> >>>>>>>>>> It’s going to be a lot of work building out the tests and I
> >> think
> >>>>>>>>>> we
> >>>>>>>> need a plan for that. My thoughts:
> >>>>>>>>>>
> >>>>>>>>>> 1. Step one is to make it easy to write meaningful automated
> tests
> >>>>>>>>>> and
> >>>>>>>> establish a clear pattern.
> >>>>>>>>>> 2. Step two is to start writing tests starting from the
> >>>>>>>> most-used/easiest to beak pieces and work out from there.
> >>>>>>>>>> 3. Once the pattern is established, any new pieces MUST have
> >> testing
> >>>>>>>> coverage.
> >>>>>>>>>> 4. When fixing bugs, attention should be paid to adding testing
> >> for
> >>>>>>>> that component.
> >>>>>>>>>> 5. When a pull request comes in on a piece which does not have
> >> unit
> >>>>>>>> test, a test must be written before accepting the pull request.
> The
> >>>>>>>> test
> >>>>>>>> does not need to be written by the requester. Before examining the
> >>>>>>>> request,
> >>>>>>>> the test should be written to pass for expected behavior and fail
> >> for
> >>>>>>>> the
> >>>>>>>> bug that the pull request is attempting to fix (assuming the pull
> >>>>>>>> request
> >>>>>>>> is to fix a bug).
> >>>>>>>>>>
> >>>>>>>>>> Thoughts?
> >>>>>>>>>> Harbs
> >>>>>>>>>>
> >>>>>>>>>> P.S. I’m thinking of coming to the US in late December/early
> >>>>>>>>>> January.
> >>>>>>>> I would be interested in getting together for a hacking session
> with
> >>>>>>>> folks
> >>>>>>>> who are available.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=
> >> https%3A%2F%2Fw
> >>>>>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&
> >> data=02%7C01%7C%7C
> >>>>>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> >> cee1%7
> >>>>>>>>>> C0%7C0%7C636454304479378270&sdata=
> cnyBtnVycpg3Aa7Hfel3dkAlez2m7M
> >> 0uSl3
> >>>>>>>>>> f0ezbSZY%3D&reserved=0+
> >>>>>>>> request+builder+plugin
> >>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=
> >> https%3A%2F%2Fwiki.
> >>>>>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> >> 5d35%7Cfa7b
> >>>>>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
> >> sdata=eqe6ec%
> >>>>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
> >>>>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
> >>>>>>>>>>
> >>>>>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=
> >> https%3A%2F%2Fd
> >>>>>>>>>> ocs.travis-ci.com%2Fuser%2Fgui-and-headless-browsers%
> >> 2F&data=02%7C01%
> >>>>>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%
> >> 7Cfa7b1b5a7b34438794aed2c178dec
> >>>>>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%
> >> 2FdOBmWJUsF7SylIQtMQQpAeOhw
> >>>>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
> >>>>>>>>
> >>>>>>>> https://na01.safelinks.protection.outlook.com/?url=
> >> https%3A%2F%2Fdocs.t
> >>>>>>>> ravis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%
> >> 7C01%7C%7C67
> >>>>>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> >> cee1%7C0%7
> >>>>>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03s
> >> jOytutB4
> >>>>>>>> rGCY%3D&reserved=0>
> >>>>>>>>>>
> >>>>>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=
> >> https%3A%2F%2Fs
> >>>>>>>>>> aucelabs.com%2Fproducts%2Fintegrations&data=02%7C01%
> >> 7C%7C67a58a332b69
> >>>>>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> >> cee1%7C0%7C0%7C636
> >>>>>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%
> >> 2FNXSRVOQho9qUW1mqdw%3
> >>>>>>>>>> D&reserved=0
> >>>>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=
> >> https%3A%2F%2Fsau
> >>>>>>>>>> celabs.com%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> >> 5d35%7Cfa
> >>>>>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
> >> sdata=xpG
> >>>>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
> >>>>>>>> products/integrations>
> >>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >>
> >
> >
> > --
> >
> > Piotr Zarzycki
> >
> > Patreon: *https://www.patreon.com/piotrzarzycki
> > <https://www.patreon.com/piotrzarzycki>*
>
>
--
Piotr Zarzycki
Patreon: *https://www.patreon.com/piotrzarzycki
<https://www.patreon.com/piotrzarzycki>*
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
Yeah. I noticed your commit.
I’ll try to be more careful in the future.
> On Nov 6, 2017, at 6:57 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>
> Please be aware that in Ant, there are two build paths, one if you have
> AIR_HOME pointing to an AIR SDK and another for JS Only. If you make
> changes to the builds, you might need to test the build both with and
> without AIR_HOME/PLAYERLGLOBAL_HOME. And look for failures from the CI
> after you commit something. The apacheflexbuild server is slow so it can
> take hours for the build with your changes to get through the queue.
>
> Thanks,
> -Alex
>
> On 11/6/17, 1:42 AM, "Harbs" <harbs.lists@gmail.com <ma...@gmail.com>> wrote:
>
>> Well, Josh reversed the order of the arguments, and I think that makes
>> sense. The string is optional and it was at the beginning. It’s now the
>> optional last argument and all the arguments are as typed as they can be.
>> (compare arguments are of type Object.)
>>
>> I just merged the testing branch into develop, so tests can be added on
>> the develop branch. The js testing is not functional yet, but the tests
>> can be written in a platform agnostic way.
>>
>> More to come…
>>
>> Harbs
>>
>>> On Nov 6, 2017, at 12:51 AM, Piotr Zarzycki <pi...@gmail.com>
>>> wrote:
>>>
>>> My guess is that, cause this approach emulates JUnit implementation.
>>> Let's
>>> say assertFalse - you can pass there one argument Boolean or two
>>> arguments
>>> Boolean and message. In AS3 there is no ability to create two functions
>>> with the same name. But in Java method overloading probably exists, so
>>> in
>>> JUnit there is probably:
>>>
>>> assertFalse(Boolean param);
>>> assertFalse(Boolean param, String);
>>>
>>> Summarize everything - I see only some AS3 language restriction as an
>>> reason.
>>>
>>> Piotr
>>>
>>>
>>>
>>> 2017-11-05 23:14 GMT+01:00 Harbs <ha...@gmail.com>:
>>>
>>>> BTW:
>>>>
>>>> Was there a good reason FlexUnit uses the spread operator for assert
>>>> arguments instead of typed arguments?
>>>>
>>>>> On Nov 5, 2017, at 11:36 PM, Harbs <ha...@gmail.com> wrote:
>>>>>
>>>>> Some progress on the unit tests:
>>>>>
>>>>> I got the Core tests to run on the swf side when doing a full build.
>>>>> In
>>>> order to do that I needed to move the running of the tests until after
>>>> Basic finished building so the test app could be built.
>>>>>
>>>>> We’re going to need to do something similar with the Maven build.
>>>>>
>>>>> The Testing project is compiling and is usable (on a basic level) too.
>>>> Tests should run with both a SWF and JS target. The SWF target uses
>>>> FlexUnit, while the JS target uses Josh’s implementation.
>>>>>
>>>>> I also migrated the Core tests to use the new project and that is
>>>> building as well.
>>>>>
>>>>> I still need to add the bits that Greg wrote and wire up some JS
>>>>> tests.
>>>>>
>>>>> Hopefully I’ll make some more progress tomorrow.
>>>>>
>>>>> Harbs
>>>>>
>>>>>> On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID>
>>>> wrote:
>>>>>>
>>>>>> Those warnings do not seem to affect the success of the tests on SWF
>>>>>> for
>>>>>> me.
>>>>>>
>>>>>> IMO, the big challenge is in capturing test results so it can be
>>>> reported
>>>>>> in a CI dashboard. Having a GUI display results in Flash or in a
>>>> browser
>>>>>> is a great first step, but I believe going from there to getting
>>>> Selenium
>>>>>> to report individual test failures may be trickier.
>>>>>>
>>>>>> Peter tried to write up Mustella a long time ago. It might need
>>>> updating.
>>>>>> See:
>>>>>>
>>>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki>
>>>>>> .apache.org <http://apache.org/>%2Fconfluence%2Fdisplay%2FFLEX%2FMustella%2BOverview&data=0
>>>>>> 2%7C01%7C%7Ce5b49c8e3cb349c5ca7508d524faa9d3%7Cfa7b1b5a7b34438794aed2c
>>>>>> 178decee1%7C0%7C0%7C636455581363121312&sdata=IN6067QM%2B7vPr4WAwrYXNjR
>>>>>> gNhAL6EdInCJUhfGyLjI%3D&reserved=0
>>>>>>
>>>>>> The advantage of declarative testing languages is that you get to
>>>>>> constrain the kinds of things that a test can do, and abstract when
>>>>>> the
>>>>>> test harness does it. It is much harder to interrupt sequential
>>>>>> lines
>>>> of
>>>>>> ActionScript/JavaScript.
>>>>>>
>>>>>> HTH,
>>>>>> -Alex
>>>>>>
>>>>>> On 11/4/17, 3:13 PM, "Harbs" <harbs.lists@gmail.com <ma...@gmail.com>> wrote:
>>>>>>
>>>>>>> It looks like FlexUnit is currently broken.
>>>>>>>
>>>>>>> I’m getting the following output when trying to run the FlexUnit
>>>>>>> tests
>>>> on
>>>>>>> Core and Basic:
>>>>>>>
>>>>>>> [mxmlc]
>>>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>>>> FlexUnit4/target/flexuni
>>>>>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>>>>>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description
>>>>>>> in
>>>> the
>>>>>>> SWC
>>>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>>>> FlexUnit4/target/flexuni
>>>>>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>>>>>> [mxmlc]
>>>>>>> [mxmlc]
>>>>>>> [mxmlc]
>>>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>>>> FlexUnit4/target/flexuni
>>>>>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>>>>>> mx.utils.StringUtil depended on by
>>>>>>> flexunit.framework.AsyncTestHelper
>>>> in
>>>>>>> the SWC
>>>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>>>> FlexUnit4/target/flexuni
>>>>>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>>>>>>
>>>>>>> Has anyone looked at this yet?
>>>>>>>
>>>>>>> Harbs
>>>>>>>
>>>>>>>> On Nov 4, 2017, at 10:45 PM, Harbs <harbs.lists@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>
>>>>>>>> Thanks! That looks very useful.
>>>>>>>>
>>>>>>>> I started work on a feature/testing branch. I’ll try to merge what
>>>>>>>> you
>>>>>>>> did into what Josh did. I’m going to try to get the existing Core
>>>> and/or
>>>>>>>> Basic tests working in both swf and node js output tomorrow. We’ll
>>>>>>>> see
>>>>>>>> how well I do… ;-)
>>>>>>>>
>>>>>>>> We might need different configs for different testing environments,
>>>> but
>>>>>>>> I’ll see if we can keep the divergence of the environs is minimal
>>>>>>>> as
>>>>>>>> possible.
>>>>>>>>
>>>>>>>> Once I get the basics worked out, I’ll likely have an idea what
>>>>>>>> others
>>>>>>>> can help with. Let’s see if we can whip the Royale tests into
>>>>>>>> shape.
>>>> :-)
>>>>>>>>
>>>>>>>> Harbs
>>>>>>>>
>>>>>>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <greg.dove@gmail.com <ma...@gmail.com>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Harbs, I don't know if it still works since changes to Royale,
>>>>>>>>> but I
>>>>>>>>> had
>>>>>>>>> something rudimentary for cross-target unit testing quite shortly
>>>> after
>>>>>>>>> working on the reflection support last year, in fact that was my
>>>>>>>>> primary
>>>>>>>>> reason for choosing to focus on reflection at the time.
>>>>>>>>>
>>>>>>>>> It was visual output only (i.e. just to look at the test results
>>>>>>>>> output in
>>>>>>>>> the browser) and the goal was to get some compatibility with
>>>>>>>>> flexunit
>>>>>>>>> (so
>>>>>>>>> that the same swf-based flexunit tests that were running in the
>>>>>>>>> build
>>>>>>>>> -e..g
>>>>>>>>> BinaryData tests- could be run visually side by side between swf
>>>>>>>>> and
>>>>>>>>> js).
>>>>>>>>> I also added a new TestVariance metadata to account for known
>>>>>>>>> (expected?)
>>>>>>>>> variation between swf and js. This was needed to cover (for
>>>>>>>>> example)
>>>>>>>>> things
>>>>>>>>> like testing Reflection classes themselves, where the results can
>>>>>>>>> be
>>>>>>>>> different between the targets based on the 'native' base classes
>>>> (e..g
>>>>>>>>> EventDispatcher inheritance chain).
>>>>>>>>>
>>>>>>>>> Some of that code might also be useful to harvest for what you are
>>>>>>>>> doing,
>>>>>>>>> I'm not sure...
>>>>>>>>>
>>>>>>>>> It's here:
>>>>>>>>>
>>>>>>>>> https://na01.safelinks.protection.outlook.com/?url= <https://na01.safelinks.protection.outlook.com/?url=>
>>>> https%3A%2F%2Fgithub.
>>>>>>>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%
>>>> 2Fmanualtests%2FUnitTests&da
>>>>>>>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d1
>>>> 5d35%7Cfa7b1b5a7b34438794aed
>>>>>>>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=
>>>> DX9gXiLqUoFQizp6xrakUwxjl
>>>>>>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <harbs.lists@gmail.com <ma...@gmail.com>>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Awesome!
>>>>>>>>>>
>>>>>>>>>> I assume that this means I can code the code to the Royale repo
>>>> under
>>>>>>>>>> Apache 2. Right?
>>>>>>>>>>
>>>>>>>>>> Harbs
>>>>>>>>>>
>>>>>>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala
>>>>>>>>>>> <joshtynjala@apache.org <ma...@apache.org>>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Please feel free to use my test runner. I'm happy to commit it
>>>>>>>>>>> somewhere
>>>>>>>>>> to make it official Apache code, if necessary.
>>>>>>>>>>>
>>>>>>>>>>> - Josh
>>>>>>>>>>>
>>>>>>>>>>> On 2017-11-03 04:19, Harbs <harbs.lists@gmail.com <ma...@gmail.com>> wrote:
>>>>>>>>>>>> One topic which keeps coming up is better test coverage for
>>>> Royale.
>>>>>>>>>>>>
>>>>>>>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good
>>>> test
>>>>>>>>>> coverage for confidence of quality and confidence that we don’t
>>>>>>>>>> introduce
>>>>>>>>>> recessive bugs.
>>>>>>>>>>>> 2. It’s really hard to accept Github pull requests without
>>>>>>>>>>>> examining
>>>>>>>>>> the code VERY well that it does not introduce recessive bugs. CI
>>>>>>>>>> which runs
>>>>>>>>>> automated tests could give a preliminary test on pull requests to
>>>>>>>>>> ensure
>>>>>>>>>> that they don’t break anything. If the pull requests do break
>>>>>>>>>> something,
>>>>>>>>>> it allows the requester to fix the problem with confidence
>>>>>>>>>> without
>>>>>>>>>> taking
>>>>>>>>>> others’ time.
>>>>>>>>>>>>
>>>>>>>>>>>> I think we need to break up testing into pieces and figure out
>>>>>>>>>>>> a
>>>>>>>>>> strategy to implement automated tests in a way that they are
>>>>>>>>>> maintainable.
>>>>>>>>>>>>
>>>>>>>>>>>> Some points:
>>>>>>>>>>>> 1. I think integration into something like Travis would be very
>>>>>>>>>>>> helpful.
>>>>>>>>>>>> 2. I think there’s a Jenkins plugin for building pull
>>>>>>>>>>>> requests.
>>>>>>>>>>>> Not
>>>>>>>>>> sure how useful it is.[1]
>>>>>>>>>>>> 3. Josh has created a Node.js-compatible test-runner
>>>>>>>>>>>> architecture
>>>>>>>>>>>> which
>>>>>>>>>> could be useful for unit tests on parts of the framework which
>>>>>>>>>> don’t rely
>>>>>>>>>> on browser features. (i.e. models and the like) He mentioned the
>>>>>>>>>> possibility of donating it, and I think that would be a very
>>>>>>>>>> useful
>>>>>>>>>> feature.
>>>>>>>>>>>> 4. For UI and integration tests, there seem to be some pretty
>>>>>>>>>>>> cool
>>>>>>>>>> integrations using Selenium.[2][3]
>>>>>>>>>>>> 5. I think the main testing effort needs to be using JS and
>>>>>>>>>>>> something
>>>>>>>>>> like Josh’s testing framework for non-UI pieces and some
>>>> easy-to-use
>>>>>>>>>> Selenium integration for visual pieces and integration tests.
>>>>>>>>>>>> 6. We probably also want some API endpoints we can test off of
>>>>>>>>>>>> for
>>>>>>>>>> integration testing.
>>>>>>>>>>>>
>>>>>>>>>>>> I’m willing to invest time into this.
>>>>>>>>>>>>
>>>>>>>>>>>> It’s going to be a lot of work building out the tests and I
>>>> think
>>>>>>>>>>>> we
>>>>>>>>>> need a plan for that. My thoughts:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. Step one is to make it easy to write meaningful automated
>>>>>>>>>>>> tests
>>>>>>>>>>>> and
>>>>>>>>>> establish a clear pattern.
>>>>>>>>>>>> 2. Step two is to start writing tests starting from the
>>>>>>>>>> most-used/easiest to beak pieces and work out from there.
>>>>>>>>>>>> 3. Once the pattern is established, any new pieces MUST have
>>>> testing
>>>>>>>>>> coverage.
>>>>>>>>>>>> 4. When fixing bugs, attention should be paid to adding testing
>>>> for
>>>>>>>>>> that component.
>>>>>>>>>>>> 5. When a pull request comes in on a piece which does not have
>>>> unit
>>>>>>>>>> test, a test must be written before accepting the pull request.
>>>>>>>>>> The
>>>>>>>>>> test
>>>>>>>>>> does not need to be written by the requester. Before examining
>>>>>>>>>> the
>>>>>>>>>> request,
>>>>>>>>>> the test should be written to pass for expected behavior and fail
>>>> for
>>>>>>>>>> the
>>>>>>>>>> bug that the pull request is attempting to fix (assuming the pull
>>>>>>>>>> request
>>>>>>>>>> is to fix a bug).
>>>>>>>>>>>>
>>>>>>>>>>>> Thoughts?
>>>>>>>>>>>> Harbs
>>>>>>>>>>>>
>>>>>>>>>>>> P.S. I’m thinking of coming to the US in late December/early
>>>>>>>>>>>> January.
>>>>>>>>>> I would be interested in getting together for a hacking session
>>>>>>>>>> with
>>>>>>>>>> folks
>>>>>>>>>> who are available.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url= <https://na01.safelinks.protection.outlook.com/?url=>
>>>> https%3A%2F%2Fw
>>>>>>>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&
>>>> data=02%7C01%7C%7C
>>>>>>>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
>>>> cee1%7
>>>>>>>>>>>>
>>>>>>>>>>>> C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M
>>>> 0uSl3
>>>>>>>>>>>> f0ezbSZY%3D&reserved=0+
>>>>>>>>>> request+builder+plugin
>>>>>>>>>> <https://na01.safelinks.protection.outlook.com/?url= <https://na01.safelinks.protection.outlook.com/?url=>
>>>> https%3A%2F%2Fwiki.
>>>>>>>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
>>>> 5d35%7Cfa7b
>>>>>>>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
>>>> sdata=eqe6ec%
>>>>>>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
>>>>>>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>>>>>>>>
>>>>>>>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url= <https://na01.safelinks.protection.outlook.com/?url=>
>>>> https%3A%2F%2Fd
>>>>>>>>>>>> ocs.travis-ci.com <http://ocs.travis-ci.com/>%2Fuser%2Fgui-and-headless-browsers%
>>>> 2F&data=02%7C01%
>>>>>>>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%
>>>> 7Cfa7b1b5a7b34438794aed2c178dec
>>>>>>>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%
>>>> 2FdOBmWJUsF7SylIQtMQQpAeOhw
>>>>>>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
>>>>>>>>>>
>>>>>>>>>> https://na01.safelinks.protection.outlook.com/?url= <https://na01.safelinks.protection.outlook.com/?url=>
>>>> https%3A%2F%2Fdocs.t
>>>>>>>>>> ravis-ci.com <http://ravis-ci.com/>%2Fuser%2Fgui-and-headless-browsers%2F&data=02%
>>>> 7C01%7C%7C67
>>>>>>>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
>>>> cee1%7C0%7
>>>>>>>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03s
>>>> jOytutB4
>>>>>>>>>> rGCY%3D&reserved=0>
>>>>>>>>>>>>
>>>>>>>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url= <https://na01.safelinks.protection.outlook.com/?url=>
>>>> https%3A%2F%2Fs
>>>>>>>>>>>> aucelabs.com <http://aucelabs.com/>%2Fproducts%2Fintegrations&data=02%7C01%
>>>> 7C%7C67a58a332b69
>>>>>>>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
>>>> cee1%7C0%7C0%7C636
>>>>>>>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%
>>>> 2FNXSRVOQho9qUW1mqdw%3
>>>>>>>>>>>> D&reserved=0
>>>>>>>>>>>> <https://na01.safelinks.protection.outlook.com/?url= <https://na01.safelinks.protection.outlook.com/?url=>
>>>> https%3A%2F%2Fsau
>>>>>>>>>>>> celabs.com <http://celabs.com/>%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
>>>> 5d35%7Cfa
>>>>>>>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
>>>> sdata=xpG
>>>>>>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
>>>>>>>>>> products/integrations>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> Piotr Zarzycki
>>>
>>> Patreon:
>>> *https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.pat <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.pat>
>>> reon.com <http://reon.com/>%2Fpiotrzarzycki&data=02%7C01%7C%7Ce5b49c8e3cb349c5ca7508d524faa9
>>> d3%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636455581363121312&sdata=
>>> QBPbqXZLLgir1Cf93t%2BlEadVLXo3eq83ifD4wjwBYAs%3D&reserved=0
>>>
>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.pat <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.pat>
>>> reon.com <http://reon.com/>%2Fpiotrzarzycki&data=02%7C01%7C%7Ce5b49c8e3cb349c5ca7508d524faa9
>>> d3%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636455581363121312&sdata=
>>> QBPbqXZLLgir1Cf93t%2BlEadVLXo3eq83ifD4wjwBYAs%3D&reserved=0>*
Re: Unit Tests et. al.
Posted by Alex Harui <ah...@adobe.com.INVALID>.
Please be aware that in Ant, there are two build paths, one if you have
AIR_HOME pointing to an AIR SDK and another for JS Only. If you make
changes to the builds, you might need to test the build both with and
without AIR_HOME/PLAYERLGLOBAL_HOME. And look for failures from the CI
after you commit something. The apacheflexbuild server is slow so it can
take hours for the build with your changes to get through the queue.
Thanks,
-Alex
On 11/6/17, 1:42 AM, "Harbs" <ha...@gmail.com> wrote:
>Well, Josh reversed the order of the arguments, and I think that makes
>sense. The string is optional and it was at the beginning. It’s now the
>optional last argument and all the arguments are as typed as they can be.
>(compare arguments are of type Object.)
>
>I just merged the testing branch into develop, so tests can be added on
>the develop branch. The js testing is not functional yet, but the tests
>can be written in a platform agnostic way.
>
>More to come…
>
>Harbs
>
>> On Nov 6, 2017, at 12:51 AM, Piotr Zarzycki <pi...@gmail.com>
>>wrote:
>>
>> My guess is that, cause this approach emulates JUnit implementation.
>>Let's
>> say assertFalse - you can pass there one argument Boolean or two
>>arguments
>> Boolean and message. In AS3 there is no ability to create two functions
>> with the same name. But in Java method overloading probably exists, so
>>in
>> JUnit there is probably:
>>
>> assertFalse(Boolean param);
>> assertFalse(Boolean param, String);
>>
>> Summarize everything - I see only some AS3 language restriction as an
>> reason.
>>
>> Piotr
>>
>>
>>
>> 2017-11-05 23:14 GMT+01:00 Harbs <ha...@gmail.com>:
>>
>>> BTW:
>>>
>>> Was there a good reason FlexUnit uses the spread operator for assert
>>> arguments instead of typed arguments?
>>>
>>>> On Nov 5, 2017, at 11:36 PM, Harbs <ha...@gmail.com> wrote:
>>>>
>>>> Some progress on the unit tests:
>>>>
>>>> I got the Core tests to run on the swf side when doing a full build.
>>>>In
>>> order to do that I needed to move the running of the tests until after
>>> Basic finished building so the test app could be built.
>>>>
>>>> We’re going to need to do something similar with the Maven build.
>>>>
>>>> The Testing project is compiling and is usable (on a basic level) too.
>>> Tests should run with both a SWF and JS target. The SWF target uses
>>> FlexUnit, while the JS target uses Josh’s implementation.
>>>>
>>>> I also migrated the Core tests to use the new project and that is
>>> building as well.
>>>>
>>>> I still need to add the bits that Greg wrote and wire up some JS
>>>>tests.
>>>>
>>>> Hopefully I’ll make some more progress tomorrow.
>>>>
>>>> Harbs
>>>>
>>>>> On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID>
>>> wrote:
>>>>>
>>>>> Those warnings do not seem to affect the success of the tests on SWF
>>>>>for
>>>>> me.
>>>>>
>>>>> IMO, the big challenge is in capturing test results so it can be
>>> reported
>>>>> in a CI dashboard. Having a GUI display results in Flash or in a
>>> browser
>>>>> is a great first step, but I believe going from there to getting
>>> Selenium
>>>>> to report individual test failures may be trickier.
>>>>>
>>>>> Peter tried to write up Mustella a long time ago. It might need
>>> updating.
>>>>> See:
>>>>>
>>>>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki
>>>>>.apache.org%2Fconfluence%2Fdisplay%2FFLEX%2FMustella%2BOverview&data=0
>>>>>2%7C01%7C%7Ce5b49c8e3cb349c5ca7508d524faa9d3%7Cfa7b1b5a7b34438794aed2c
>>>>>178decee1%7C0%7C0%7C636455581363121312&sdata=IN6067QM%2B7vPr4WAwrYXNjR
>>>>>gNhAL6EdInCJUhfGyLjI%3D&reserved=0
>>>>>
>>>>> The advantage of declarative testing languages is that you get to
>>>>> constrain the kinds of things that a test can do, and abstract when
>>>>>the
>>>>> test harness does it. It is much harder to interrupt sequential
>>>>>lines
>>> of
>>>>> ActionScript/JavaScript.
>>>>>
>>>>> HTH,
>>>>> -Alex
>>>>>
>>>>> On 11/4/17, 3:13 PM, "Harbs" <ha...@gmail.com> wrote:
>>>>>
>>>>>> It looks like FlexUnit is currently broken.
>>>>>>
>>>>>> I’m getting the following output when trying to run the FlexUnit
>>>>>>tests
>>> on
>>>>>> Core and Basic:
>>>>>>
>>>>>> [mxmlc]
>>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>>> FlexUnit4/target/flexuni
>>>>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>>>>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description
>>>>>>in
>>> the
>>>>>> SWC
>>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>>> FlexUnit4/target/flexuni
>>>>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>>>>> [mxmlc]
>>>>>> [mxmlc]
>>>>>> [mxmlc]
>>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>>> FlexUnit4/target/flexuni
>>>>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>>>>> mx.utils.StringUtil depended on by
>>>>>>flexunit.framework.AsyncTestHelper
>>> in
>>>>>> the SWC
>>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>>> FlexUnit4/target/flexuni
>>>>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>>>>>
>>>>>> Has anyone looked at this yet?
>>>>>>
>>>>>> Harbs
>>>>>>
>>>>>>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
>>>>>>>
>>>>>>> Thanks! That looks very useful.
>>>>>>>
>>>>>>> I started work on a feature/testing branch. I’ll try to merge what
>>>>>>>you
>>>>>>> did into what Josh did. I’m going to try to get the existing Core
>>> and/or
>>>>>>> Basic tests working in both swf and node js output tomorrow. We’ll
>>>>>>>see
>>>>>>> how well I do… ;-)
>>>>>>>
>>>>>>> We might need different configs for different testing environments,
>>> but
>>>>>>> I’ll see if we can keep the divergence of the environs is minimal
>>>>>>>as
>>>>>>> possible.
>>>>>>>
>>>>>>> Once I get the basics worked out, I’ll likely have an idea what
>>>>>>>others
>>>>>>> can help with. Let’s see if we can whip the Royale tests into
>>>>>>>shape.
>>> :-)
>>>>>>>
>>>>>>> Harbs
>>>>>>>
>>>>>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>> Harbs, I don't know if it still works since changes to Royale,
>>>>>>>>but I
>>>>>>>> had
>>>>>>>> something rudimentary for cross-target unit testing quite shortly
>>> after
>>>>>>>> working on the reflection support last year, in fact that was my
>>>>>>>> primary
>>>>>>>> reason for choosing to focus on reflection at the time.
>>>>>>>>
>>>>>>>> It was visual output only (i.e. just to look at the test results
>>>>>>>> output in
>>>>>>>> the browser) and the goal was to get some compatibility with
>>>>>>>>flexunit
>>>>>>>> (so
>>>>>>>> that the same swf-based flexunit tests that were running in the
>>>>>>>>build
>>>>>>>> -e..g
>>>>>>>> BinaryData tests- could be run visually side by side between swf
>>>>>>>>and
>>>>>>>> js).
>>>>>>>> I also added a new TestVariance metadata to account for known
>>>>>>>> (expected?)
>>>>>>>> variation between swf and js. This was needed to cover (for
>>>>>>>>example)
>>>>>>>> things
>>>>>>>> like testing Reflection classes themselves, where the results can
>>>>>>>>be
>>>>>>>> different between the targets based on the 'native' base classes
>>> (e..g
>>>>>>>> EventDispatcher inheritance chain).
>>>>>>>>
>>>>>>>> Some of that code might also be useful to harvest for what you are
>>>>>>>> doing,
>>>>>>>> I'm not sure...
>>>>>>>>
>>>>>>>> It's here:
>>>>>>>>
>>>>>>>> https://na01.safelinks.protection.outlook.com/?url=
>>> https%3A%2F%2Fgithub.
>>>>>>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%
>>> 2Fmanualtests%2FUnitTests&da
>>>>>>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d1
>>> 5d35%7Cfa7b1b5a7b34438794aed
>>>>>>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=
>>> DX9gXiLqUoFQizp6xrakUwxjl
>>>>>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com>
>>>>>>>>wrote:
>>>>>>>>
>>>>>>>>> Awesome!
>>>>>>>>>
>>>>>>>>> I assume that this means I can code the code to the Royale repo
>>> under
>>>>>>>>> Apache 2. Right?
>>>>>>>>>
>>>>>>>>> Harbs
>>>>>>>>>
>>>>>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala
>>>>>>>>>><jo...@apache.org>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Please feel free to use my test runner. I'm happy to commit it
>>>>>>>>>> somewhere
>>>>>>>>> to make it official Apache code, if necessary.
>>>>>>>>>>
>>>>>>>>>> - Josh
>>>>>>>>>>
>>>>>>>>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>>>>>>>>>>> One topic which keeps coming up is better test coverage for
>>> Royale.
>>>>>>>>>>>
>>>>>>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good
>>> test
>>>>>>>>> coverage for confidence of quality and confidence that we don’t
>>>>>>>>> introduce
>>>>>>>>> recessive bugs.
>>>>>>>>>>> 2. It’s really hard to accept Github pull requests without
>>>>>>>>>>> examining
>>>>>>>>> the code VERY well that it does not introduce recessive bugs. CI
>>>>>>>>> which runs
>>>>>>>>> automated tests could give a preliminary test on pull requests to
>>>>>>>>> ensure
>>>>>>>>> that they don’t break anything. If the pull requests do break
>>>>>>>>> something,
>>>>>>>>> it allows the requester to fix the problem with confidence
>>>>>>>>>without
>>>>>>>>> taking
>>>>>>>>> others’ time.
>>>>>>>>>>>
>>>>>>>>>>> I think we need to break up testing into pieces and figure out
>>>>>>>>>>>a
>>>>>>>>> strategy to implement automated tests in a way that they are
>>>>>>>>> maintainable.
>>>>>>>>>>>
>>>>>>>>>>> Some points:
>>>>>>>>>>> 1. I think integration into something like Travis would be very
>>>>>>>>>>> helpful.
>>>>>>>>>>> 2. I think there’s a Jenkins plugin for building pull
>>>>>>>>>>>requests.
>>>>>>>>>>> Not
>>>>>>>>> sure how useful it is.[1]
>>>>>>>>>>> 3. Josh has created a Node.js-compatible test-runner
>>>>>>>>>>>architecture
>>>>>>>>>>> which
>>>>>>>>> could be useful for unit tests on parts of the framework which
>>>>>>>>> don’t rely
>>>>>>>>> on browser features. (i.e. models and the like) He mentioned the
>>>>>>>>> possibility of donating it, and I think that would be a very
>>>>>>>>>useful
>>>>>>>>> feature.
>>>>>>>>>>> 4. For UI and integration tests, there seem to be some pretty
>>>>>>>>>>>cool
>>>>>>>>> integrations using Selenium.[2][3]
>>>>>>>>>>> 5. I think the main testing effort needs to be using JS and
>>>>>>>>>>> something
>>>>>>>>> like Josh’s testing framework for non-UI pieces and some
>>> easy-to-use
>>>>>>>>> Selenium integration for visual pieces and integration tests.
>>>>>>>>>>> 6. We probably also want some API endpoints we can test off of
>>>>>>>>>>>for
>>>>>>>>> integration testing.
>>>>>>>>>>>
>>>>>>>>>>> I’m willing to invest time into this.
>>>>>>>>>>>
>>>>>>>>>>> It’s going to be a lot of work building out the tests and I
>>> think
>>>>>>>>>>> we
>>>>>>>>> need a plan for that. My thoughts:
>>>>>>>>>>>
>>>>>>>>>>> 1. Step one is to make it easy to write meaningful automated
>>>>>>>>>>>tests
>>>>>>>>>>> and
>>>>>>>>> establish a clear pattern.
>>>>>>>>>>> 2. Step two is to start writing tests starting from the
>>>>>>>>> most-used/easiest to beak pieces and work out from there.
>>>>>>>>>>> 3. Once the pattern is established, any new pieces MUST have
>>> testing
>>>>>>>>> coverage.
>>>>>>>>>>> 4. When fixing bugs, attention should be paid to adding testing
>>> for
>>>>>>>>> that component.
>>>>>>>>>>> 5. When a pull request comes in on a piece which does not have
>>> unit
>>>>>>>>> test, a test must be written before accepting the pull request.
>>>>>>>>>The
>>>>>>>>> test
>>>>>>>>> does not need to be written by the requester. Before examining
>>>>>>>>>the
>>>>>>>>> request,
>>>>>>>>> the test should be written to pass for expected behavior and fail
>>> for
>>>>>>>>> the
>>>>>>>>> bug that the pull request is attempting to fix (assuming the pull
>>>>>>>>> request
>>>>>>>>> is to fix a bug).
>>>>>>>>>>>
>>>>>>>>>>> Thoughts?
>>>>>>>>>>> Harbs
>>>>>>>>>>>
>>>>>>>>>>> P.S. I’m thinking of coming to the US in late December/early
>>>>>>>>>>> January.
>>>>>>>>> I would be interested in getting together for a hacking session
>>>>>>>>>with
>>>>>>>>> folks
>>>>>>>>> who are available.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=
>>> https%3A%2F%2Fw
>>>>>>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&
>>> data=02%7C01%7C%7C
>>>>>>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
>>> cee1%7
>>>>>>>>>>>
>>>>>>>>>>>C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M
>>> 0uSl3
>>>>>>>>>>> f0ezbSZY%3D&reserved=0+
>>>>>>>>> request+builder+plugin
>>>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=
>>> https%3A%2F%2Fwiki.
>>>>>>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
>>> 5d35%7Cfa7b
>>>>>>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
>>> sdata=eqe6ec%
>>>>>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
>>>>>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>>>>>>>
>>>>>>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=
>>> https%3A%2F%2Fd
>>>>>>>>>>> ocs.travis-ci.com%2Fuser%2Fgui-and-headless-browsers%
>>> 2F&data=02%7C01%
>>>>>>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%
>>> 7Cfa7b1b5a7b34438794aed2c178dec
>>>>>>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%
>>> 2FdOBmWJUsF7SylIQtMQQpAeOhw
>>>>>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
>>>>>>>>>
>>>>>>>>> https://na01.safelinks.protection.outlook.com/?url=
>>> https%3A%2F%2Fdocs.t
>>>>>>>>> ravis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%
>>> 7C01%7C%7C67
>>>>>>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
>>> cee1%7C0%7
>>>>>>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03s
>>> jOytutB4
>>>>>>>>> rGCY%3D&reserved=0>
>>>>>>>>>>>
>>>>>>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=
>>> https%3A%2F%2Fs
>>>>>>>>>>> aucelabs.com%2Fproducts%2Fintegrations&data=02%7C01%
>>> 7C%7C67a58a332b69
>>>>>>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
>>> cee1%7C0%7C0%7C636
>>>>>>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%
>>> 2FNXSRVOQho9qUW1mqdw%3
>>>>>>>>>>> D&reserved=0
>>>>>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=
>>> https%3A%2F%2Fsau
>>>>>>>>>>> celabs.com%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
>>> 5d35%7Cfa
>>>>>>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
>>> sdata=xpG
>>>>>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
>>>>>>>>> products/integrations>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>> --
>>
>> Piotr Zarzycki
>>
>> Patreon:
>>*https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.pat
>>reon.com%2Fpiotrzarzycki&data=02%7C01%7C%7Ce5b49c8e3cb349c5ca7508d524faa9
>>d3%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636455581363121312&sdata=
>>QBPbqXZLLgir1Cf93t%2BlEadVLXo3eq83ifD4wjwBYAs%3D&reserved=0
>>
>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.pat
>>reon.com%2Fpiotrzarzycki&data=02%7C01%7C%7Ce5b49c8e3cb349c5ca7508d524faa9
>>d3%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636455581363121312&sdata=
>>QBPbqXZLLgir1Cf93t%2BlEadVLXo3eq83ifD4wjwBYAs%3D&reserved=0>*
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
Well, Josh reversed the order of the arguments, and I think that makes sense. The string is optional and it was at the beginning. It’s now the optional last argument and all the arguments are as typed as they can be. (compare arguments are of type Object.)
I just merged the testing branch into develop, so tests can be added on the develop branch. The js testing is not functional yet, but the tests can be written in a platform agnostic way.
More to come…
Harbs
> On Nov 6, 2017, at 12:51 AM, Piotr Zarzycki <pi...@gmail.com> wrote:
>
> My guess is that, cause this approach emulates JUnit implementation. Let's
> say assertFalse - you can pass there one argument Boolean or two arguments
> Boolean and message. In AS3 there is no ability to create two functions
> with the same name. But in Java method overloading probably exists, so in
> JUnit there is probably:
>
> assertFalse(Boolean param);
> assertFalse(Boolean param, String);
>
> Summarize everything - I see only some AS3 language restriction as an
> reason.
>
> Piotr
>
>
>
> 2017-11-05 23:14 GMT+01:00 Harbs <ha...@gmail.com>:
>
>> BTW:
>>
>> Was there a good reason FlexUnit uses the spread operator for assert
>> arguments instead of typed arguments?
>>
>>> On Nov 5, 2017, at 11:36 PM, Harbs <ha...@gmail.com> wrote:
>>>
>>> Some progress on the unit tests:
>>>
>>> I got the Core tests to run on the swf side when doing a full build. In
>> order to do that I needed to move the running of the tests until after
>> Basic finished building so the test app could be built.
>>>
>>> We’re going to need to do something similar with the Maven build.
>>>
>>> The Testing project is compiling and is usable (on a basic level) too.
>> Tests should run with both a SWF and JS target. The SWF target uses
>> FlexUnit, while the JS target uses Josh’s implementation.
>>>
>>> I also migrated the Core tests to use the new project and that is
>> building as well.
>>>
>>> I still need to add the bits that Greg wrote and wire up some JS tests.
>>>
>>> Hopefully I’ll make some more progress tomorrow.
>>>
>>> Harbs
>>>
>>>> On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID>
>> wrote:
>>>>
>>>> Those warnings do not seem to affect the success of the tests on SWF for
>>>> me.
>>>>
>>>> IMO, the big challenge is in capturing test results so it can be
>> reported
>>>> in a CI dashboard. Having a GUI display results in Flash or in a
>> browser
>>>> is a great first step, but I believe going from there to getting
>> Selenium
>>>> to report individual test failures may be trickier.
>>>>
>>>> Peter tried to write up Mustella a long time ago. It might need
>> updating.
>>>> See:
>>>> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
>>>>
>>>> The advantage of declarative testing languages is that you get to
>>>> constrain the kinds of things that a test can do, and abstract when the
>>>> test harness does it. It is much harder to interrupt sequential lines
>> of
>>>> ActionScript/JavaScript.
>>>>
>>>> HTH,
>>>> -Alex
>>>>
>>>> On 11/4/17, 3:13 PM, "Harbs" <ha...@gmail.com> wrote:
>>>>
>>>>> It looks like FlexUnit is currently broken.
>>>>>
>>>>> I’m getting the following output when trying to run the FlexUnit tests
>> on
>>>>> Core and Basic:
>>>>>
>>>>> [mxmlc]
>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>> FlexUnit4/target/flexuni
>>>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>>>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in
>> the
>>>>> SWC
>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>> FlexUnit4/target/flexuni
>>>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>>>> [mxmlc]
>>>>> [mxmlc]
>>>>> [mxmlc]
>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>> FlexUnit4/target/flexuni
>>>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>>>> mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper
>> in
>>>>> the SWC
>>>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
>> FlexUnit4/target/flexuni
>>>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>>>>
>>>>> Has anyone looked at this yet?
>>>>>
>>>>> Harbs
>>>>>
>>>>>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
>>>>>>
>>>>>> Thanks! That looks very useful.
>>>>>>
>>>>>> I started work on a feature/testing branch. I’ll try to merge what you
>>>>>> did into what Josh did. I’m going to try to get the existing Core
>> and/or
>>>>>> Basic tests working in both swf and node js output tomorrow. We’ll see
>>>>>> how well I do… ;-)
>>>>>>
>>>>>> We might need different configs for different testing environments,
>> but
>>>>>> I’ll see if we can keep the divergence of the environs is minimal as
>>>>>> possible.
>>>>>>
>>>>>> Once I get the basics worked out, I’ll likely have an idea what others
>>>>>> can help with. Let’s see if we can whip the Royale tests into shape.
>> :-)
>>>>>>
>>>>>> Harbs
>>>>>>
>>>>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
>>>>>>>
>>>>>>> Harbs, I don't know if it still works since changes to Royale, but I
>>>>>>> had
>>>>>>> something rudimentary for cross-target unit testing quite shortly
>> after
>>>>>>> working on the reflection support last year, in fact that was my
>>>>>>> primary
>>>>>>> reason for choosing to focus on reflection at the time.
>>>>>>>
>>>>>>> It was visual output only (i.e. just to look at the test results
>>>>>>> output in
>>>>>>> the browser) and the goal was to get some compatibility with flexunit
>>>>>>> (so
>>>>>>> that the same swf-based flexunit tests that were running in the build
>>>>>>> -e..g
>>>>>>> BinaryData tests- could be run visually side by side between swf and
>>>>>>> js).
>>>>>>> I also added a new TestVariance metadata to account for known
>>>>>>> (expected?)
>>>>>>> variation between swf and js. This was needed to cover (for example)
>>>>>>> things
>>>>>>> like testing Reflection classes themselves, where the results can be
>>>>>>> different between the targets based on the 'native' base classes
>> (e..g
>>>>>>> EventDispatcher inheritance chain).
>>>>>>>
>>>>>>> Some of that code might also be useful to harvest for what you are
>>>>>>> doing,
>>>>>>> I'm not sure...
>>>>>>>
>>>>>>> It's here:
>>>>>>>
>>>>>>> https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fgithub.
>>>>>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%
>> 2Fmanualtests%2FUnitTests&da
>>>>>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d1
>> 5d35%7Cfa7b1b5a7b34438794aed
>>>>>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=
>> DX9gXiLqUoFQizp6xrakUwxjl
>>>>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Awesome!
>>>>>>>>
>>>>>>>> I assume that this means I can code the code to the Royale repo
>> under
>>>>>>>> Apache 2. Right?
>>>>>>>>
>>>>>>>> Harbs
>>>>>>>>
>>>>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Please feel free to use my test runner. I'm happy to commit it
>>>>>>>>> somewhere
>>>>>>>> to make it official Apache code, if necessary.
>>>>>>>>>
>>>>>>>>> - Josh
>>>>>>>>>
>>>>>>>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>>>>>>>>>> One topic which keeps coming up is better test coverage for
>> Royale.
>>>>>>>>>>
>>>>>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good
>> test
>>>>>>>> coverage for confidence of quality and confidence that we don’t
>>>>>>>> introduce
>>>>>>>> recessive bugs.
>>>>>>>>>> 2. It’s really hard to accept Github pull requests without
>>>>>>>>>> examining
>>>>>>>> the code VERY well that it does not introduce recessive bugs. CI
>>>>>>>> which runs
>>>>>>>> automated tests could give a preliminary test on pull requests to
>>>>>>>> ensure
>>>>>>>> that they don’t break anything. If the pull requests do break
>>>>>>>> something,
>>>>>>>> it allows the requester to fix the problem with confidence without
>>>>>>>> taking
>>>>>>>> others’ time.
>>>>>>>>>>
>>>>>>>>>> I think we need to break up testing into pieces and figure out a
>>>>>>>> strategy to implement automated tests in a way that they are
>>>>>>>> maintainable.
>>>>>>>>>>
>>>>>>>>>> Some points:
>>>>>>>>>> 1. I think integration into something like Travis would be very
>>>>>>>>>> helpful.
>>>>>>>>>> 2. I think there’s a Jenkins plugin for building pull requests.
>>>>>>>>>> Not
>>>>>>>> sure how useful it is.[1]
>>>>>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
>>>>>>>>>> which
>>>>>>>> could be useful for unit tests on parts of the framework which
>>>>>>>> don’t rely
>>>>>>>> on browser features. (i.e. models and the like) He mentioned the
>>>>>>>> possibility of donating it, and I think that would be a very useful
>>>>>>>> feature.
>>>>>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>>>>>> integrations using Selenium.[2][3]
>>>>>>>>>> 5. I think the main testing effort needs to be using JS and
>>>>>>>>>> something
>>>>>>>> like Josh’s testing framework for non-UI pieces and some
>> easy-to-use
>>>>>>>> Selenium integration for visual pieces and integration tests.
>>>>>>>>>> 6. We probably also want some API endpoints we can test off of for
>>>>>>>> integration testing.
>>>>>>>>>>
>>>>>>>>>> I’m willing to invest time into this.
>>>>>>>>>>
>>>>>>>>>> It’s going to be a lot of work building out the tests and I
>> think
>>>>>>>>>> we
>>>>>>>> need a plan for that. My thoughts:
>>>>>>>>>>
>>>>>>>>>> 1. Step one is to make it easy to write meaningful automated tests
>>>>>>>>>> and
>>>>>>>> establish a clear pattern.
>>>>>>>>>> 2. Step two is to start writing tests starting from the
>>>>>>>> most-used/easiest to beak pieces and work out from there.
>>>>>>>>>> 3. Once the pattern is established, any new pieces MUST have
>> testing
>>>>>>>> coverage.
>>>>>>>>>> 4. When fixing bugs, attention should be paid to adding testing
>> for
>>>>>>>> that component.
>>>>>>>>>> 5. When a pull request comes in on a piece which does not have
>> unit
>>>>>>>> test, a test must be written before accepting the pull request. The
>>>>>>>> test
>>>>>>>> does not need to be written by the requester. Before examining the
>>>>>>>> request,
>>>>>>>> the test should be written to pass for expected behavior and fail
>> for
>>>>>>>> the
>>>>>>>> bug that the pull request is attempting to fix (assuming the pull
>>>>>>>> request
>>>>>>>> is to fix a bug).
>>>>>>>>>>
>>>>>>>>>> Thoughts?
>>>>>>>>>> Harbs
>>>>>>>>>>
>>>>>>>>>> P.S. I’m thinking of coming to the US in late December/early
>>>>>>>>>> January.
>>>>>>>> I would be interested in getting together for a hacking session with
>>>>>>>> folks
>>>>>>>> who are available.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fw
>>>>>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&
>> data=02%7C01%7C%7C
>>>>>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
>> cee1%7
>>>>>>>>>> C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M
>> 0uSl3
>>>>>>>>>> f0ezbSZY%3D&reserved=0+
>>>>>>>> request+builder+plugin
>>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fwiki.
>>>>>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
>> 5d35%7Cfa7b
>>>>>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
>> sdata=eqe6ec%
>>>>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
>>>>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>>>>>>
>>>>>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fd
>>>>>>>>>> ocs.travis-ci.com%2Fuser%2Fgui-and-headless-browsers%
>> 2F&data=02%7C01%
>>>>>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%
>> 7Cfa7b1b5a7b34438794aed2c178dec
>>>>>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%
>> 2FdOBmWJUsF7SylIQtMQQpAeOhw
>>>>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
>>>>>>>>
>>>>>>>> https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fdocs.t
>>>>>>>> ravis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%
>> 7C01%7C%7C67
>>>>>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
>> cee1%7C0%7
>>>>>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03s
>> jOytutB4
>>>>>>>> rGCY%3D&reserved=0>
>>>>>>>>>>
>>>>>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fs
>>>>>>>>>> aucelabs.com%2Fproducts%2Fintegrations&data=02%7C01%
>> 7C%7C67a58a332b69
>>>>>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
>> cee1%7C0%7C0%7C636
>>>>>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%
>> 2FNXSRVOQho9qUW1mqdw%3
>>>>>>>>>> D&reserved=0
>>>>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=
>> https%3A%2F%2Fsau
>>>>>>>>>> celabs.com%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
>> 5d35%7Cfa
>>>>>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
>> sdata=xpG
>>>>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
>>>>>>>> products/integrations>
>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
> --
>
> Piotr Zarzycki
>
> Patreon: *https://www.patreon.com/piotrzarzycki
> <https://www.patreon.com/piotrzarzycki>*
Re: Unit Tests et. al.
Posted by Piotr Zarzycki <pi...@gmail.com>.
My guess is that, cause this approach emulates JUnit implementation. Let's
say assertFalse - you can pass there one argument Boolean or two arguments
Boolean and message. In AS3 there is no ability to create two functions
with the same name. But in Java method overloading probably exists, so in
JUnit there is probably:
assertFalse(Boolean param);
assertFalse(Boolean param, String);
Summarize everything - I see only some AS3 language restriction as an
reason.
Piotr
2017-11-05 23:14 GMT+01:00 Harbs <ha...@gmail.com>:
> BTW:
>
> Was there a good reason FlexUnit uses the spread operator for assert
> arguments instead of typed arguments?
>
> > On Nov 5, 2017, at 11:36 PM, Harbs <ha...@gmail.com> wrote:
> >
> > Some progress on the unit tests:
> >
> > I got the Core tests to run on the swf side when doing a full build. In
> order to do that I needed to move the running of the tests until after
> Basic finished building so the test app could be built.
> >
> > We’re going to need to do something similar with the Maven build.
> >
> > The Testing project is compiling and is usable (on a basic level) too.
> Tests should run with both a SWF and JS target. The SWF target uses
> FlexUnit, while the JS target uses Josh’s implementation.
> >
> > I also migrated the Core tests to use the new project and that is
> building as well.
> >
> > I still need to add the bits that Greg wrote and wire up some JS tests.
> >
> > Hopefully I’ll make some more progress tomorrow.
> >
> > Harbs
> >
> >> On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID>
> wrote:
> >>
> >> Those warnings do not seem to affect the success of the tests on SWF for
> >> me.
> >>
> >> IMO, the big challenge is in capturing test results so it can be
> reported
> >> in a CI dashboard. Having a GUI display results in Flash or in a
> browser
> >> is a great first step, but I believe going from there to getting
> Selenium
> >> to report individual test failures may be trickier.
> >>
> >> Peter tried to write up Mustella a long time ago. It might need
> updating.
> >> See:
> >> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
> >>
> >> The advantage of declarative testing languages is that you get to
> >> constrain the kinds of things that a test can do, and abstract when the
> >> test harness does it. It is much harder to interrupt sequential lines
> of
> >> ActionScript/JavaScript.
> >>
> >> HTH,
> >> -Alex
> >>
> >> On 11/4/17, 3:13 PM, "Harbs" <ha...@gmail.com> wrote:
> >>
> >>> It looks like FlexUnit is currently broken.
> >>>
> >>> I’m getting the following output when trying to run the FlexUnit tests
> on
> >>> Core and Basic:
> >>>
> >>> [mxmlc]
> >>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
> >>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in
> the
> >>> SWC
> >>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
> >>> [mxmlc]
> >>> [mxmlc]
> >>> [mxmlc]
> >>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
> >>> mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper
> in
> >>> the SWC
> >>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
> >>>
> >>> Has anyone looked at this yet?
> >>>
> >>> Harbs
> >>>
> >>>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
> >>>>
> >>>> Thanks! That looks very useful.
> >>>>
> >>>> I started work on a feature/testing branch. I’ll try to merge what you
> >>>> did into what Josh did. I’m going to try to get the existing Core
> and/or
> >>>> Basic tests working in both swf and node js output tomorrow. We’ll see
> >>>> how well I do… ;-)
> >>>>
> >>>> We might need different configs for different testing environments,
> but
> >>>> I’ll see if we can keep the divergence of the environs is minimal as
> >>>> possible.
> >>>>
> >>>> Once I get the basics worked out, I’ll likely have an idea what others
> >>>> can help with. Let’s see if we can whip the Royale tests into shape.
> :-)
> >>>>
> >>>> Harbs
> >>>>
> >>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
> >>>>>
> >>>>> Harbs, I don't know if it still works since changes to Royale, but I
> >>>>> had
> >>>>> something rudimentary for cross-target unit testing quite shortly
> after
> >>>>> working on the reflection support last year, in fact that was my
> >>>>> primary
> >>>>> reason for choosing to focus on reflection at the time.
> >>>>>
> >>>>> It was visual output only (i.e. just to look at the test results
> >>>>> output in
> >>>>> the browser) and the goal was to get some compatibility with flexunit
> >>>>> (so
> >>>>> that the same swf-based flexunit tests that were running in the build
> >>>>> -e..g
> >>>>> BinaryData tests- could be run visually side by side between swf and
> >>>>> js).
> >>>>> I also added a new TestVariance metadata to account for known
> >>>>> (expected?)
> >>>>> variation between swf and js. This was needed to cover (for example)
> >>>>> things
> >>>>> like testing Reflection classes themselves, where the results can be
> >>>>> different between the targets based on the 'native' base classes
> (e..g
> >>>>> EventDispatcher inheritance chain).
> >>>>>
> >>>>> Some of that code might also be useful to harvest for what you are
> >>>>> doing,
> >>>>> I'm not sure...
> >>>>>
> >>>>> It's here:
> >>>>>
> >>>>> https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fgithub.
> >>>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%
> 2Fmanualtests%2FUnitTests&da
> >>>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> 5d35%7Cfa7b1b5a7b34438794aed
> >>>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=
> DX9gXiLqUoFQizp6xrakUwxjl
> >>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com> wrote:
> >>>>>
> >>>>>> Awesome!
> >>>>>>
> >>>>>> I assume that this means I can code the code to the Royale repo
> under
> >>>>>> Apache 2. Right?
> >>>>>>
> >>>>>> Harbs
> >>>>>>
> >>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org>
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>> Please feel free to use my test runner. I'm happy to commit it
> >>>>>>> somewhere
> >>>>>> to make it official Apache code, if necessary.
> >>>>>>>
> >>>>>>> - Josh
> >>>>>>>
> >>>>>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
> >>>>>>>> One topic which keeps coming up is better test coverage for
> Royale.
> >>>>>>>>
> >>>>>>>> I think this is becoming a critical issue for a few reasons:
> >>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good
> test
> >>>>>> coverage for confidence of quality and confidence that we don’t
> >>>>>> introduce
> >>>>>> recessive bugs.
> >>>>>>>> 2. It’s really hard to accept Github pull requests without
> >>>>>>>> examining
> >>>>>> the code VERY well that it does not introduce recessive bugs. CI
> >>>>>> which runs
> >>>>>> automated tests could give a preliminary test on pull requests to
> >>>>>> ensure
> >>>>>> that they don’t break anything. If the pull requests do break
> >>>>>> something,
> >>>>>> it allows the requester to fix the problem with confidence without
> >>>>>> taking
> >>>>>> others’ time.
> >>>>>>>>
> >>>>>>>> I think we need to break up testing into pieces and figure out a
> >>>>>> strategy to implement automated tests in a way that they are
> >>>>>> maintainable.
> >>>>>>>>
> >>>>>>>> Some points:
> >>>>>>>> 1. I think integration into something like Travis would be very
> >>>>>>>> helpful.
> >>>>>>>> 2. I think there’s a Jenkins plugin for building pull requests.
> >>>>>>>> Not
> >>>>>> sure how useful it is.[1]
> >>>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
> >>>>>>>> which
> >>>>>> could be useful for unit tests on parts of the framework which
> >>>>>> don’t rely
> >>>>>> on browser features. (i.e. models and the like) He mentioned the
> >>>>>> possibility of donating it, and I think that would be a very useful
> >>>>>> feature.
> >>>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
> >>>>>> integrations using Selenium.[2][3]
> >>>>>>>> 5. I think the main testing effort needs to be using JS and
> >>>>>>>> something
> >>>>>> like Josh’s testing framework for non-UI pieces and some
> easy-to-use
> >>>>>> Selenium integration for visual pieces and integration tests.
> >>>>>>>> 6. We probably also want some API endpoints we can test off of for
> >>>>>> integration testing.
> >>>>>>>>
> >>>>>>>> I’m willing to invest time into this.
> >>>>>>>>
> >>>>>>>> It’s going to be a lot of work building out the tests and I
> think
> >>>>>>>> we
> >>>>>> need a plan for that. My thoughts:
> >>>>>>>>
> >>>>>>>> 1. Step one is to make it easy to write meaningful automated tests
> >>>>>>>> and
> >>>>>> establish a clear pattern.
> >>>>>>>> 2. Step two is to start writing tests starting from the
> >>>>>> most-used/easiest to beak pieces and work out from there.
> >>>>>>>> 3. Once the pattern is established, any new pieces MUST have
> testing
> >>>>>> coverage.
> >>>>>>>> 4. When fixing bugs, attention should be paid to adding testing
> for
> >>>>>> that component.
> >>>>>>>> 5. When a pull request comes in on a piece which does not have
> unit
> >>>>>> test, a test must be written before accepting the pull request. The
> >>>>>> test
> >>>>>> does not need to be written by the requester. Before examining the
> >>>>>> request,
> >>>>>> the test should be written to pass for expected behavior and fail
> for
> >>>>>> the
> >>>>>> bug that the pull request is attempting to fix (assuming the pull
> >>>>>> request
> >>>>>> is to fix a bug).
> >>>>>>>>
> >>>>>>>> Thoughts?
> >>>>>>>> Harbs
> >>>>>>>>
> >>>>>>>> P.S. I’m thinking of coming to the US in late December/early
> >>>>>>>> January.
> >>>>>> I would be interested in getting together for a hacking session with
> >>>>>> folks
> >>>>>> who are available.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fw
> >>>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&
> data=02%7C01%7C%7C
> >>>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7
> >>>>>>>> C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M
> 0uSl3
> >>>>>>>> f0ezbSZY%3D&reserved=0+
> >>>>>> request+builder+plugin
> >>>>>> <https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fwiki.
> >>>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> 5d35%7Cfa7b
> >>>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
> sdata=eqe6ec%
> >>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
> >>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
> >>>>>>>>
> >>>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fd
> >>>>>>>> ocs.travis-ci.com%2Fuser%2Fgui-and-headless-browsers%
> 2F&data=02%7C01%
> >>>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%
> 7Cfa7b1b5a7b34438794aed2c178dec
> >>>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%
> 2FdOBmWJUsF7SylIQtMQQpAeOhw
> >>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
> >>>>>>
> >>>>>> https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fdocs.t
> >>>>>> ravis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%
> 7C01%7C%7C67
> >>>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7C0%7
> >>>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03s
> jOytutB4
> >>>>>> rGCY%3D&reserved=0>
> >>>>>>>>
> >>>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fs
> >>>>>>>> aucelabs.com%2Fproducts%2Fintegrations&data=02%7C01%
> 7C%7C67a58a332b69
> >>>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7C0%7C0%7C636
> >>>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%
> 2FNXSRVOQho9qUW1mqdw%3
> >>>>>>>> D&reserved=0
> >>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fsau
> >>>>>>>> celabs.com%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> 5d35%7Cfa
> >>>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
> sdata=xpG
> >>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
> >>>>>> products/integrations>
> >>>>>>
> >>>>>>
> >>>>
> >>>
> >>
> >
>
>
--
Piotr Zarzycki
Patreon: *https://www.patreon.com/piotrzarzycki
<https://www.patreon.com/piotrzarzycki>*
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
BTW:
Was there a good reason FlexUnit uses the spread operator for assert arguments instead of typed arguments?
> On Nov 5, 2017, at 11:36 PM, Harbs <ha...@gmail.com> wrote:
>
> Some progress on the unit tests:
>
> I got the Core tests to run on the swf side when doing a full build. In order to do that I needed to move the running of the tests until after Basic finished building so the test app could be built.
>
> We’re going to need to do something similar with the Maven build.
>
> The Testing project is compiling and is usable (on a basic level) too. Tests should run with both a SWF and JS target. The SWF target uses FlexUnit, while the JS target uses Josh’s implementation.
>
> I also migrated the Core tests to use the new project and that is building as well.
>
> I still need to add the bits that Greg wrote and wire up some JS tests.
>
> Hopefully I’ll make some more progress tomorrow.
>
> Harbs
>
>> On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>>
>> Those warnings do not seem to affect the success of the tests on SWF for
>> me.
>>
>> IMO, the big challenge is in capturing test results so it can be reported
>> in a CI dashboard. Having a GUI display results in Flash or in a browser
>> is a great first step, but I believe going from there to getting Selenium
>> to report individual test failures may be trickier.
>>
>> Peter tried to write up Mustella a long time ago. It might need updating.
>> See:
>> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
>>
>> The advantage of declarative testing languages is that you get to
>> constrain the kinds of things that a test can do, and abstract when the
>> test harness does it. It is much harder to interrupt sequential lines of
>> ActionScript/JavaScript.
>>
>> HTH,
>> -Alex
>>
>> On 11/4/17, 3:13 PM, "Harbs" <ha...@gmail.com> wrote:
>>
>>> It looks like FlexUnit is currently broken.
>>>
>>> I’m getting the following output when trying to run the FlexUnit tests on
>>> Core and Basic:
>>>
>>> [mxmlc]
>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in the
>>> SWC
>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>> [mxmlc]
>>> [mxmlc]
>>> [mxmlc]
>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>> mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper in
>>> the SWC
>>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>>
>>> Has anyone looked at this yet?
>>>
>>> Harbs
>>>
>>>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
>>>>
>>>> Thanks! That looks very useful.
>>>>
>>>> I started work on a feature/testing branch. I’ll try to merge what you
>>>> did into what Josh did. I’m going to try to get the existing Core and/or
>>>> Basic tests working in both swf and node js output tomorrow. We’ll see
>>>> how well I do… ;-)
>>>>
>>>> We might need different configs for different testing environments, but
>>>> I’ll see if we can keep the divergence of the environs is minimal as
>>>> possible.
>>>>
>>>> Once I get the basics worked out, I’ll likely have an idea what others
>>>> can help with. Let’s see if we can whip the Royale tests into shape. :-)
>>>>
>>>> Harbs
>>>>
>>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
>>>>>
>>>>> Harbs, I don't know if it still works since changes to Royale, but I
>>>>> had
>>>>> something rudimentary for cross-target unit testing quite shortly after
>>>>> working on the reflection support last year, in fact that was my
>>>>> primary
>>>>> reason for choosing to focus on reflection at the time.
>>>>>
>>>>> It was visual output only (i.e. just to look at the test results
>>>>> output in
>>>>> the browser) and the goal was to get some compatibility with flexunit
>>>>> (so
>>>>> that the same swf-based flexunit tests that were running in the build
>>>>> -e..g
>>>>> BinaryData tests- could be run visually side by side between swf and
>>>>> js).
>>>>> I also added a new TestVariance metadata to account for known
>>>>> (expected?)
>>>>> variation between swf and js. This was needed to cover (for example)
>>>>> things
>>>>> like testing Reflection classes themselves, where the results can be
>>>>> different between the targets based on the 'native' base classes (e..g
>>>>> EventDispatcher inheritance chain).
>>>>>
>>>>> Some of that code might also be useful to harvest for what you are
>>>>> doing,
>>>>> I'm not sure...
>>>>>
>>>>> It's here:
>>>>>
>>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
>>>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%2Fmanualtests%2FUnitTests&da
>>>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed
>>>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=DX9gXiLqUoFQizp6xrakUwxjl
>>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com> wrote:
>>>>>
>>>>>> Awesome!
>>>>>>
>>>>>> I assume that this means I can code the code to the Royale repo under
>>>>>> Apache 2. Right?
>>>>>>
>>>>>> Harbs
>>>>>>
>>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Please feel free to use my test runner. I'm happy to commit it
>>>>>>> somewhere
>>>>>> to make it official Apache code, if necessary.
>>>>>>>
>>>>>>> - Josh
>>>>>>>
>>>>>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>>>>>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>>>>>
>>>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>>>>> coverage for confidence of quality and confidence that we don’t
>>>>>> introduce
>>>>>> recessive bugs.
>>>>>>>> 2. It’s really hard to accept Github pull requests without
>>>>>>>> examining
>>>>>> the code VERY well that it does not introduce recessive bugs. CI
>>>>>> which runs
>>>>>> automated tests could give a preliminary test on pull requests to
>>>>>> ensure
>>>>>> that they don’t break anything. If the pull requests do break
>>>>>> something,
>>>>>> it allows the requester to fix the problem with confidence without
>>>>>> taking
>>>>>> others’ time.
>>>>>>>>
>>>>>>>> I think we need to break up testing into pieces and figure out a
>>>>>> strategy to implement automated tests in a way that they are
>>>>>> maintainable.
>>>>>>>>
>>>>>>>> Some points:
>>>>>>>> 1. I think integration into something like Travis would be very
>>>>>>>> helpful.
>>>>>>>> 2. I think there’s a Jenkins plugin for building pull requests.
>>>>>>>> Not
>>>>>> sure how useful it is.[1]
>>>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
>>>>>>>> which
>>>>>> could be useful for unit tests on parts of the framework which
>>>>>> don’t rely
>>>>>> on browser features. (i.e. models and the like) He mentioned the
>>>>>> possibility of donating it, and I think that would be a very useful
>>>>>> feature.
>>>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>>>> integrations using Selenium.[2][3]
>>>>>>>> 5. I think the main testing effort needs to be using JS and
>>>>>>>> something
>>>>>> like Josh’s testing framework for non-UI pieces and some easy-to-use
>>>>>> Selenium integration for visual pieces and integration tests.
>>>>>>>> 6. We probably also want some API endpoints we can test off of for
>>>>>> integration testing.
>>>>>>>>
>>>>>>>> I’m willing to invest time into this.
>>>>>>>>
>>>>>>>> It’s going to be a lot of work building out the tests and I think
>>>>>>>> we
>>>>>> need a plan for that. My thoughts:
>>>>>>>>
>>>>>>>> 1. Step one is to make it easy to write meaningful automated tests
>>>>>>>> and
>>>>>> establish a clear pattern.
>>>>>>>> 2. Step two is to start writing tests starting from the
>>>>>> most-used/easiest to beak pieces and work out from there.
>>>>>>>> 3. Once the pattern is established, any new pieces MUST have testing
>>>>>> coverage.
>>>>>>>> 4. When fixing bugs, attention should be paid to adding testing for
>>>>>> that component.
>>>>>>>> 5. When a pull request comes in on a piece which does not have unit
>>>>>> test, a test must be written before accepting the pull request. The
>>>>>> test
>>>>>> does not need to be written by the requester. Before examining the
>>>>>> request,
>>>>>> the test should be written to pass for expected behavior and fail for
>>>>>> the
>>>>>> bug that the pull request is attempting to fix (assuming the pull
>>>>>> request
>>>>>> is to fix a bug).
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>> Harbs
>>>>>>>>
>>>>>>>> P.S. I’m thinking of coming to the US in late December/early
>>>>>>>> January.
>>>>>> I would be interested in getting together for a hacking session with
>>>>>> folks
>>>>>> who are available.
>>>>>>>>
>>>>>>>>
>>>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>>>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&data=02%7C01%7C%7C
>>>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7
>>>>>>>> C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M0uSl3
>>>>>>>> f0ezbSZY%3D&reserved=0+
>>>>>> request+builder+plugin
>>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.
>>>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b
>>>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=eqe6ec%
>>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
>>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>>>>
>>>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fd
>>>>>>>> ocs.travis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%
>>>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178dec
>>>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw
>>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
>>>>>>
>>>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.t
>>>>>> ravis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C67
>>>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7
>>>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03sjOytutB4
>>>>>> rGCY%3D&reserved=0>
>>>>>>>>
>>>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs
>>>>>>>> aucelabs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C67a58a332b69
>>>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636
>>>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%2FNXSRVOQho9qUW1mqdw%3
>>>>>>>> D&reserved=0
>>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsau
>>>>>>>> celabs.com%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa
>>>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=xpG
>>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
>>>>>> products/integrations>
>>>>>>
>>>>>>
>>>>
>>>
>>
>
Re: Unit Tests et. al.
Posted by Piotr Zarzycki <pi...@gmail.com>.
Harbs,
You can leave it to me Maven part, but I have to admit that I will be able
to take look into that probably at the end of the upcoming week. Still need
to finish release of Installer and SDK. If you find the time please raise
for me issue on GitHub, unless you would like to touch Maven.
Thanks! Piotr
2017-11-05 22:36 GMT+01:00 Harbs <ha...@gmail.com>:
> Some progress on the unit tests:
>
> I got the Core tests to run on the swf side when doing a full build. In
> order to do that I needed to move the running of the tests until after
> Basic finished building so the test app could be built.
>
> We’re going to need to do something similar with the Maven build.
>
> The Testing project is compiling and is usable (on a basic level) too.
> Tests should run with both a SWF and JS target. The SWF target uses
> FlexUnit, while the JS target uses Josh’s implementation.
>
> I also migrated the Core tests to use the new project and that is building
> as well.
>
> I still need to add the bits that Greg wrote and wire up some JS tests.
>
> Hopefully I’ll make some more progress tomorrow.
>
> Harbs
>
> > On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID> wrote:
> >
> > Those warnings do not seem to affect the success of the tests on SWF for
> > me.
> >
> > IMO, the big challenge is in capturing test results so it can be reported
> > in a CI dashboard. Having a GUI display results in Flash or in a browser
> > is a great first step, but I believe going from there to getting Selenium
> > to report individual test failures may be trickier.
> >
> > Peter tried to write up Mustella a long time ago. It might need updating.
> > See:
> > https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
> >
> > The advantage of declarative testing languages is that you get to
> > constrain the kinds of things that a test can do, and abstract when the
> > test harness does it. It is much harder to interrupt sequential lines of
> > ActionScript/JavaScript.
> >
> > HTH,
> > -Alex
> >
> > On 11/4/17, 3:13 PM, "Harbs" <ha...@gmail.com> wrote:
> >
> >> It looks like FlexUnit is currently broken.
> >>
> >> I’m getting the following output when trying to run the FlexUnit tests
> on
> >> Core and Basic:
> >>
> >> [mxmlc]
> >> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
> >> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in
> the
> >> SWC
> >> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >> t-4.3.0-20140410-as3_4.12.0.swc could not be found
> >> [mxmlc]
> >> [mxmlc]
> >> [mxmlc]
> >> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
> >> mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper
> in
> >> the SWC
> >> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/
> FlexUnit4/target/flexuni
> >> t-4.3.0-20140410-as3_4.12.0.swc could not be found
> >>
> >> Has anyone looked at this yet?
> >>
> >> Harbs
> >>
> >>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
> >>>
> >>> Thanks! That looks very useful.
> >>>
> >>> I started work on a feature/testing branch. I’ll try to merge what you
> >>> did into what Josh did. I’m going to try to get the existing Core
> and/or
> >>> Basic tests working in both swf and node js output tomorrow. We’ll see
> >>> how well I do… ;-)
> >>>
> >>> We might need different configs for different testing environments, but
> >>> I’ll see if we can keep the divergence of the environs is minimal as
> >>> possible.
> >>>
> >>> Once I get the basics worked out, I’ll likely have an idea what others
> >>> can help with. Let’s see if we can whip the Royale tests into shape.
> :-)
> >>>
> >>> Harbs
> >>>
> >>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
> >>>>
> >>>> Harbs, I don't know if it still works since changes to Royale, but I
> >>>> had
> >>>> something rudimentary for cross-target unit testing quite shortly
> after
> >>>> working on the reflection support last year, in fact that was my
> >>>> primary
> >>>> reason for choosing to focus on reflection at the time.
> >>>>
> >>>> It was visual output only (i.e. just to look at the test results
> >>>> output in
> >>>> the browser) and the goal was to get some compatibility with flexunit
> >>>> (so
> >>>> that the same swf-based flexunit tests that were running in the build
> >>>> -e..g
> >>>> BinaryData tests- could be run visually side by side between swf and
> >>>> js).
> >>>> I also added a new TestVariance metadata to account for known
> >>>> (expected?)
> >>>> variation between swf and js. This was needed to cover (for example)
> >>>> things
> >>>> like testing Reflection classes themselves, where the results can be
> >>>> different between the targets based on the 'native' base classes (e..g
> >>>> EventDispatcher inheritance chain).
> >>>>
> >>>> Some of that code might also be useful to harvest for what you are
> >>>> doing,
> >>>> I'm not sure...
> >>>>
> >>>> It's here:
> >>>>
> >>>> https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fgithub.
> >>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%
> 2Fmanualtests%2FUnitTests&da
> >>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> 5d35%7Cfa7b1b5a7b34438794aed
> >>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=
> DX9gXiLqUoFQizp6xrakUwxjl
> >>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
> >>>>
> >>>>
> >>>>
> >>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com> wrote:
> >>>>
> >>>>> Awesome!
> >>>>>
> >>>>> I assume that this means I can code the code to the Royale repo under
> >>>>> Apache 2. Right?
> >>>>>
> >>>>> Harbs
> >>>>>
> >>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org>
> >>>>>> wrote:
> >>>>>>
> >>>>>> Please feel free to use my test runner. I'm happy to commit it
> >>>>>> somewhere
> >>>>> to make it official Apache code, if necessary.
> >>>>>>
> >>>>>> - Josh
> >>>>>>
> >>>>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
> >>>>>>> One topic which keeps coming up is better test coverage for Royale.
> >>>>>>>
> >>>>>>> I think this is becoming a critical issue for a few reasons:
> >>>>>>> 1. As we get close to version 1.0 it’s necessary to have good
> test
> >>>>> coverage for confidence of quality and confidence that we don’t
> >>>>> introduce
> >>>>> recessive bugs.
> >>>>>>> 2. It’s really hard to accept Github pull requests without
> >>>>>>> examining
> >>>>> the code VERY well that it does not introduce recessive bugs. CI
> >>>>> which runs
> >>>>> automated tests could give a preliminary test on pull requests to
> >>>>> ensure
> >>>>> that they don’t break anything. If the pull requests do break
> >>>>> something,
> >>>>> it allows the requester to fix the problem with confidence without
> >>>>> taking
> >>>>> others’ time.
> >>>>>>>
> >>>>>>> I think we need to break up testing into pieces and figure out a
> >>>>> strategy to implement automated tests in a way that they are
> >>>>> maintainable.
> >>>>>>>
> >>>>>>> Some points:
> >>>>>>> 1. I think integration into something like Travis would be very
> >>>>>>> helpful.
> >>>>>>> 2. I think there’s a Jenkins plugin for building pull requests.
> >>>>>>> Not
> >>>>> sure how useful it is.[1]
> >>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
> >>>>>>> which
> >>>>> could be useful for unit tests on parts of the framework which
> >>>>> don’t rely
> >>>>> on browser features. (i.e. models and the like) He mentioned the
> >>>>> possibility of donating it, and I think that would be a very useful
> >>>>> feature.
> >>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
> >>>>> integrations using Selenium.[2][3]
> >>>>>>> 5. I think the main testing effort needs to be using JS and
> >>>>>>> something
> >>>>> like Josh’s testing framework for non-UI pieces and some
> easy-to-use
> >>>>> Selenium integration for visual pieces and integration tests.
> >>>>>>> 6. We probably also want some API endpoints we can test off of for
> >>>>> integration testing.
> >>>>>>>
> >>>>>>> I’m willing to invest time into this.
> >>>>>>>
> >>>>>>> It’s going to be a lot of work building out the tests and I think
> >>>>>>> we
> >>>>> need a plan for that. My thoughts:
> >>>>>>>
> >>>>>>> 1. Step one is to make it easy to write meaningful automated tests
> >>>>>>> and
> >>>>> establish a clear pattern.
> >>>>>>> 2. Step two is to start writing tests starting from the
> >>>>> most-used/easiest to beak pieces and work out from there.
> >>>>>>> 3. Once the pattern is established, any new pieces MUST have
> testing
> >>>>> coverage.
> >>>>>>> 4. When fixing bugs, attention should be paid to adding testing for
> >>>>> that component.
> >>>>>>> 5. When a pull request comes in on a piece which does not have unit
> >>>>> test, a test must be written before accepting the pull request. The
> >>>>> test
> >>>>> does not need to be written by the requester. Before examining the
> >>>>> request,
> >>>>> the test should be written to pass for expected behavior and fail for
> >>>>> the
> >>>>> bug that the pull request is attempting to fix (assuming the pull
> >>>>> request
> >>>>> is to fix a bug).
> >>>>>>>
> >>>>>>> Thoughts?
> >>>>>>> Harbs
> >>>>>>>
> >>>>>>> P.S. I’m thinking of coming to the US in late December/early
> >>>>>>> January.
> >>>>> I would be interested in getting together for a hacking session with
> >>>>> folks
> >>>>> who are available.
> >>>>>>>
> >>>>>>>
> >>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fw
> >>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&
> data=02%7C01%7C%7C
> >>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7
> >>>>>>> C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M
> 0uSl3
> >>>>>>> f0ezbSZY%3D&reserved=0+
> >>>>> request+builder+plugin
> >>>>> <https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fwiki.
> >>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> 5d35%7Cfa7b
> >>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
> sdata=eqe6ec%
> >>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
> >>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
> >>>>>>>
> >>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fd
> >>>>>>> ocs.travis-ci.com%2Fuser%2Fgui-and-headless-browsers%
> 2F&data=02%7C01%
> >>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%
> 7Cfa7b1b5a7b34438794aed2c178dec
> >>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%
> 2FdOBmWJUsF7SylIQtMQQpAeOhw
> >>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
> >>>>>
> >>>>> https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fdocs.t
> >>>>> ravis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%
> 7C01%7C%7C67
> >>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7C0%7
> >>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03s
> jOytutB4
> >>>>> rGCY%3D&reserved=0>
> >>>>>>>
> >>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fs
> >>>>>>> aucelabs.com%2Fproducts%2Fintegrations&data=02%7C01%
> 7C%7C67a58a332b69
> >>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178de
> cee1%7C0%7C0%7C636
> >>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%
> 2FNXSRVOQho9qUW1mqdw%3
> >>>>>>> D&reserved=0
> >>>>>>> <https://na01.safelinks.protection.outlook.com/?url=
> https%3A%2F%2Fsau
> >>>>>>> celabs.com%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d1
> 5d35%7Cfa
> >>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&
> sdata=xpG
> >>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
> >>>>> products/integrations>
> >>>>>
> >>>>>
> >>>
> >>
> >
>
>
--
Piotr Zarzycki
Patreon: *https://www.patreon.com/piotrzarzycki
<https://www.patreon.com/piotrzarzycki>*
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
Some progress on the unit tests:
I got the Core tests to run on the swf side when doing a full build. In order to do that I needed to move the running of the tests until after Basic finished building so the test app could be built.
We’re going to need to do something similar with the Maven build.
The Testing project is compiling and is usable (on a basic level) too. Tests should run with both a SWF and JS target. The SWF target uses FlexUnit, while the JS target uses Josh’s implementation.
I also migrated the Core tests to use the new project and that is building as well.
I still need to add the bits that Greg wrote and wire up some JS tests.
Hopefully I’ll make some more progress tomorrow.
Harbs
> On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>
> Those warnings do not seem to affect the success of the tests on SWF for
> me.
>
> IMO, the big challenge is in capturing test results so it can be reported
> in a CI dashboard. Having a GUI display results in Flash or in a browser
> is a great first step, but I believe going from there to getting Selenium
> to report individual test failures may be trickier.
>
> Peter tried to write up Mustella a long time ago. It might need updating.
> See:
> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
>
> The advantage of declarative testing languages is that you get to
> constrain the kinds of things that a test can do, and abstract when the
> test harness does it. It is much harder to interrupt sequential lines of
> ActionScript/JavaScript.
>
> HTH,
> -Alex
>
> On 11/4/17, 3:13 PM, "Harbs" <ha...@gmail.com> wrote:
>
>> It looks like FlexUnit is currently broken.
>>
>> I’m getting the following output when trying to run the FlexUnit tests on
>> Core and Basic:
>>
>> [mxmlc]
>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in the
>> SWC
>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>> [mxmlc]
>> [mxmlc]
>> [mxmlc]
>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>> mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper in
>> the SWC
>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>
>> Has anyone looked at this yet?
>>
>> Harbs
>>
>>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
>>>
>>> Thanks! That looks very useful.
>>>
>>> I started work on a feature/testing branch. I’ll try to merge what you
>>> did into what Josh did. I’m going to try to get the existing Core and/or
>>> Basic tests working in both swf and node js output tomorrow. We’ll see
>>> how well I do… ;-)
>>>
>>> We might need different configs for different testing environments, but
>>> I’ll see if we can keep the divergence of the environs is minimal as
>>> possible.
>>>
>>> Once I get the basics worked out, I’ll likely have an idea what others
>>> can help with. Let’s see if we can whip the Royale tests into shape. :-)
>>>
>>> Harbs
>>>
>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
>>>>
>>>> Harbs, I don't know if it still works since changes to Royale, but I
>>>> had
>>>> something rudimentary for cross-target unit testing quite shortly after
>>>> working on the reflection support last year, in fact that was my
>>>> primary
>>>> reason for choosing to focus on reflection at the time.
>>>>
>>>> It was visual output only (i.e. just to look at the test results
>>>> output in
>>>> the browser) and the goal was to get some compatibility with flexunit
>>>> (so
>>>> that the same swf-based flexunit tests that were running in the build
>>>> -e..g
>>>> BinaryData tests- could be run visually side by side between swf and
>>>> js).
>>>> I also added a new TestVariance metadata to account for known
>>>> (expected?)
>>>> variation between swf and js. This was needed to cover (for example)
>>>> things
>>>> like testing Reflection classes themselves, where the results can be
>>>> different between the targets based on the 'native' base classes (e..g
>>>> EventDispatcher inheritance chain).
>>>>
>>>> Some of that code might also be useful to harvest for what you are
>>>> doing,
>>>> I'm not sure...
>>>>
>>>> It's here:
>>>>
>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
>>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%2Fmanualtests%2FUnitTests&da
>>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed
>>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=DX9gXiLqUoFQizp6xrakUwxjl
>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
>>>>
>>>>
>>>>
>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com> wrote:
>>>>
>>>>> Awesome!
>>>>>
>>>>> I assume that this means I can code the code to the Royale repo under
>>>>> Apache 2. Right?
>>>>>
>>>>> Harbs
>>>>>
>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org>
>>>>>> wrote:
>>>>>>
>>>>>> Please feel free to use my test runner. I'm happy to commit it
>>>>>> somewhere
>>>>> to make it official Apache code, if necessary.
>>>>>>
>>>>>> - Josh
>>>>>>
>>>>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>>>>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>>>>
>>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>>>> coverage for confidence of quality and confidence that we don’t
>>>>> introduce
>>>>> recessive bugs.
>>>>>>> 2. It’s really hard to accept Github pull requests without
>>>>>>> examining
>>>>> the code VERY well that it does not introduce recessive bugs. CI
>>>>> which runs
>>>>> automated tests could give a preliminary test on pull requests to
>>>>> ensure
>>>>> that they don’t break anything. If the pull requests do break
>>>>> something,
>>>>> it allows the requester to fix the problem with confidence without
>>>>> taking
>>>>> others’ time.
>>>>>>>
>>>>>>> I think we need to break up testing into pieces and figure out a
>>>>> strategy to implement automated tests in a way that they are
>>>>> maintainable.
>>>>>>>
>>>>>>> Some points:
>>>>>>> 1. I think integration into something like Travis would be very
>>>>>>> helpful.
>>>>>>> 2. I think there’s a Jenkins plugin for building pull requests.
>>>>>>> Not
>>>>> sure how useful it is.[1]
>>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
>>>>>>> which
>>>>> could be useful for unit tests on parts of the framework which
>>>>> don’t rely
>>>>> on browser features. (i.e. models and the like) He mentioned the
>>>>> possibility of donating it, and I think that would be a very useful
>>>>> feature.
>>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>>> integrations using Selenium.[2][3]
>>>>>>> 5. I think the main testing effort needs to be using JS and
>>>>>>> something
>>>>> like Josh’s testing framework for non-UI pieces and some easy-to-use
>>>>> Selenium integration for visual pieces and integration tests.
>>>>>>> 6. We probably also want some API endpoints we can test off of for
>>>>> integration testing.
>>>>>>>
>>>>>>> I’m willing to invest time into this.
>>>>>>>
>>>>>>> It’s going to be a lot of work building out the tests and I think
>>>>>>> we
>>>>> need a plan for that. My thoughts:
>>>>>>>
>>>>>>> 1. Step one is to make it easy to write meaningful automated tests
>>>>>>> and
>>>>> establish a clear pattern.
>>>>>>> 2. Step two is to start writing tests starting from the
>>>>> most-used/easiest to beak pieces and work out from there.
>>>>>>> 3. Once the pattern is established, any new pieces MUST have testing
>>>>> coverage.
>>>>>>> 4. When fixing bugs, attention should be paid to adding testing for
>>>>> that component.
>>>>>>> 5. When a pull request comes in on a piece which does not have unit
>>>>> test, a test must be written before accepting the pull request. The
>>>>> test
>>>>> does not need to be written by the requester. Before examining the
>>>>> request,
>>>>> the test should be written to pass for expected behavior and fail for
>>>>> the
>>>>> bug that the pull request is attempting to fix (assuming the pull
>>>>> request
>>>>> is to fix a bug).
>>>>>>>
>>>>>>> Thoughts?
>>>>>>> Harbs
>>>>>>>
>>>>>>> P.S. I’m thinking of coming to the US in late December/early
>>>>>>> January.
>>>>> I would be interested in getting together for a hacking session with
>>>>> folks
>>>>> who are available.
>>>>>>>
>>>>>>>
>>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&data=02%7C01%7C%7C
>>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7
>>>>>>> C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M0uSl3
>>>>>>> f0ezbSZY%3D&reserved=0+
>>>>> request+builder+plugin
>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.
>>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b
>>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=eqe6ec%
>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>>>
>>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fd
>>>>>>> ocs.travis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%
>>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178dec
>>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw
>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
>>>>>
>>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.t
>>>>> ravis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C67
>>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7
>>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03sjOytutB4
>>>>> rGCY%3D&reserved=0>
>>>>>>>
>>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs
>>>>>>> aucelabs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C67a58a332b69
>>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636
>>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%2FNXSRVOQho9qUW1mqdw%3
>>>>>>> D&reserved=0
>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsau
>>>>>>> celabs.com%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa
>>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=xpG
>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
>>>>> products/integrations>
>>>>>
>>>>>
>>>
>>
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
“Units” can mean lots of things to different people. I’m not a purist, and I personally don’t see the value of having tiny unit tests. To me, the importance of tests is to make sure there’s good coverage and there’s insurance that regressive bugs are not introduced.
To me, there’s two types of tests that we need in Royale:
1. FlexUnit-type tests which can be run on multiple environments. The simpler the tests, the better. I see these being good fits for things like utility classes, models, platform-agnostic pieces, etc.
2. “Component tests” or “integration tests” which test that all the pieces work together as expected.
Some of the “unit type tests” might be better implemented as “integration tests”. I’d rather have more integration tests than complicated mocks and such to ensure that test cases are completely isolated.
For #1, I’m currently working on getting the existing tests working in a JS environment so we can use a single FlexUnit-type test constructs to test on both platforms.
For #2, I’m trying to figure out the best strategy. Test beads seem to me like a good way to approach it.
> On Nov 5, 2017, at 2:26 PM, Piotr Zarzycki <pi...@gmail.com> wrote:
>
> Hi Harbs,
>
> Great discussion. Just thought for Bead driven tests. - Is this what you
> are describing are "Unit" tests actually or UI specific tests - automation
> ? Cause I cannot distinguish it from discussion above.
>
> Piotr
>
>
> 2017-11-05 10:55 GMT+01:00 Harbs <harbs.lists@gmail.com <ma...@gmail.com>>:
>
>> On these two points:
>>
>>> Hmm, that would be cool if we don't need Selenium to report results.
>>> Keeping Selenium synced up is a PITA.
>>
>> For tests that are not browser-dependant, we should not need selenium.
>> Testing directly through node should be much faster besides requiring less
>> setup.
>>
>> For tests that rely on browser features, most of the testing frameworks
>> I’ve seen use selenium. Most drive it using node.js. Some use PhantomJS,
>> and I think there’s an option to run Chromium headlessly. One interesting
>> framework which does not seem to use Selenium is cypress.
>>
>> I’m still looking around. If anyone has experience with JS testing
>> frameworks, please let me know...
>>
>>>
>>> I've never looked to see how FlexUnit handles this, but I'm not
>>> clear how a test can be written in AS as:
>>>
>>> @Test
>>> Function MyTest() {
>>> SetAProperty();
>>> AssertSomeOtherProperty();
>>> }
>>
>> I’m not sure how FlexUnit does it either.
>>
>> Here are two interesting options
>> http://nightwatchjs.org/ <http://nightwatchjs.org/> <http://nightwatchjs.org/ <http://nightwatchjs.org/>>
>> https://www.cypress.io/ <https://www.cypress.io/> <https://www.cypress.io/ <https://www.cypress.io/>>
>>
>> The frameworks I’ve seen seem to have a “wait” function.
>>
>> We could probably also use event listeners.
>>
>> Here’s a nice list:
>> https://www.joecolantonio.com/2016/06/14/top-8-essential- <https://www.joecolantonio.com/2016/06/14/top-8-essential->
>> javascript-automation-frameworks/ <https://www.joecolantonio <https://www.joecolantonio/>.
>> com/2016/06/14/top-8-essential-javascript-automation-frameworks/>
>>
>> I just had an interesting idea for solving the component testing problem
>> in a Royale-specific way which might be a nice advantage over other
>> frameworks:
>>
>> Testing Beads.
>>
>> The problem with component test seem to be the following:
>> 1. Testing at the correct point in the component lifecycle.
>> 2. Being able to address specific components and their parts.
>> 3. Being able to fail-early on tests that don’t require complete loading.
>> 4. Ensuring that all tests complete — which usually means synchronous
>> execution of tests.
>>
>> Testing beads seem like they should be able to solve these problems in an
>> interesting way.
>>
>> Basically, a testing bead would be a bead which has an interface which:
>> a. Reports test passes.
>> b. reports test failures.
>> c. reports ignored test.
>> d. Reports when all tests are done.
>>
>> It would work something like this:
>> 1. A test runner/test app, would create components and add testing beads
>> to the components.
>> 2. It would retain references to the testing beads and listen for results
>> from the beads.
>> 3. The test runner would run the app.
>> 4. Each test bead would take care of running its own tests and report back
>> when done.
>> 5. Once all the test beads report success or a bead reports failure, the
>> test runner would exit with the full report.
>>
>> This would have the following advantages:
>> 1. All tests could run in parallel. This would probably speed up test runs
>> tremendously. Async operations would not block other tests from being run.
>> 2. There’s no need for the test runner to worry about life-cycles. The
>> bead would be responsible to test at the correct point in the lifecycle.
>> 3. The first test to fail could exit. Failing early could make the test
>> run much quicker when tests fail.
>> 4. You could have an option to have the test runner either report all
>> failing tests or fail early on the first one.
>> 5. Running tests should be simple with a well-defined interface, and the
>> actual tests could be as simple or as complicated as necessary.
>>
>> This seems like a very good solution from framework development.
>>
>> I’m not sure how this concept could be used for application development.
>> I guess an application developer could create a parallel testing app which
>> is the same as the app plus testing beads, but that seems awkward.
>>
>> Maybe it’s possible to use a testing CSS file which would add testing
>> beads to components for testing builds, the problem with that is that
>> there’s a requirement for code to add those beads.
>>
>> Maybe we can add special tags for adding the beads via MXML and/or
>> ActionScript which could be stripped out for non-test builds.
>>
>> Food for thought…
>> Harbs
>>
>>
>
>
> --
>
> Piotr Zarzycki
>
> Patreon: *https://www.patreon.com/piotrzarzycki <https://www.patreon.com/piotrzarzycki>
> <https://www.patreon.com/piotrzarzycki <https://www.patreon.com/piotrzarzycki>>*
Re: Unit Tests et. al.
Posted by Piotr Zarzycki <pi...@gmail.com>.
Hi Harbs,
Great discussion. Just thought for Bead driven tests. - Is this what you
are describing are "Unit" tests actually or UI specific tests - automation
? Cause I cannot distinguish it from discussion above.
Piotr
2017-11-05 10:55 GMT+01:00 Harbs <ha...@gmail.com>:
> On these two points:
>
> > Hmm, that would be cool if we don't need Selenium to report results.
> > Keeping Selenium synced up is a PITA.
>
> For tests that are not browser-dependant, we should not need selenium.
> Testing directly through node should be much faster besides requiring less
> setup.
>
> For tests that rely on browser features, most of the testing frameworks
> I’ve seen use selenium. Most drive it using node.js. Some use PhantomJS,
> and I think there’s an option to run Chromium headlessly. One interesting
> framework which does not seem to use Selenium is cypress.
>
> I’m still looking around. If anyone has experience with JS testing
> frameworks, please let me know...
>
> >
> > I've never looked to see how FlexUnit handles this, but I'm not
> > clear how a test can be written in AS as:
> >
> > @Test
> > Function MyTest() {
> > SetAProperty();
> > AssertSomeOtherProperty();
> > }
>
> I’m not sure how FlexUnit does it either.
>
> Here are two interesting options
> http://nightwatchjs.org/ <http://nightwatchjs.org/>
> https://www.cypress.io/ <https://www.cypress.io/>
>
> The frameworks I’ve seen seem to have a “wait” function.
>
> We could probably also use event listeners.
>
> Here’s a nice list:
> https://www.joecolantonio.com/2016/06/14/top-8-essential-
> javascript-automation-frameworks/ <https://www.joecolantonio.
> com/2016/06/14/top-8-essential-javascript-automation-frameworks/>
>
> I just had an interesting idea for solving the component testing problem
> in a Royale-specific way which might be a nice advantage over other
> frameworks:
>
> Testing Beads.
>
> The problem with component test seem to be the following:
> 1. Testing at the correct point in the component lifecycle.
> 2. Being able to address specific components and their parts.
> 3. Being able to fail-early on tests that don’t require complete loading.
> 4. Ensuring that all tests complete — which usually means synchronous
> execution of tests.
>
> Testing beads seem like they should be able to solve these problems in an
> interesting way.
>
> Basically, a testing bead would be a bead which has an interface which:
> a. Reports test passes.
> b. reports test failures.
> c. reports ignored test.
> d. Reports when all tests are done.
>
> It would work something like this:
> 1. A test runner/test app, would create components and add testing beads
> to the components.
> 2. It would retain references to the testing beads and listen for results
> from the beads.
> 3. The test runner would run the app.
> 4. Each test bead would take care of running its own tests and report back
> when done.
> 5. Once all the test beads report success or a bead reports failure, the
> test runner would exit with the full report.
>
> This would have the following advantages:
> 1. All tests could run in parallel. This would probably speed up test runs
> tremendously. Async operations would not block other tests from being run.
> 2. There’s no need for the test runner to worry about life-cycles. The
> bead would be responsible to test at the correct point in the lifecycle.
> 3. The first test to fail could exit. Failing early could make the test
> run much quicker when tests fail.
> 4. You could have an option to have the test runner either report all
> failing tests or fail early on the first one.
> 5. Running tests should be simple with a well-defined interface, and the
> actual tests could be as simple or as complicated as necessary.
>
> This seems like a very good solution from framework development.
>
> I’m not sure how this concept could be used for application development.
> I guess an application developer could create a parallel testing app which
> is the same as the app plus testing beads, but that seems awkward.
>
> Maybe it’s possible to use a testing CSS file which would add testing
> beads to components for testing builds, the problem with that is that
> there’s a requirement for code to add those beads.
>
> Maybe we can add special tags for adding the beads via MXML and/or
> ActionScript which could be stripped out for non-test builds.
>
> Food for thought…
> Harbs
>
>
--
Piotr Zarzycki
Patreon: *https://www.patreon.com/piotrzarzycki
<https://www.patreon.com/piotrzarzycki>*
Re: Test Beads (was Re: Unit Tests et. al.)
Posted by Harbs <ha...@gmail.com>.
> I guess we are not understanding each other.
Probably not. The best way to discuss this is likely a POC.
> If the testing language is
> AS or JS, test authors have to know how to deal with the runtime
> differences.
It depends on the tests. If the code is platform agnostic, the tests should be platform agnostic as well. If the code is platform-specific, the tests could have similar platform specific blocks. It does not seem to me like it’s a difficult problem.
> If you want to build up a test harness of tests written in AS, I would
> recommend starting with FlexUnit…
Yup. This is the low-hanging fruit here.
> If you want to run tests that require the runtime, I think Mustella might
> be a good starting point instead of trying to re-invent it.
Maybe. Once I’m finished with the unit tests, I’ll try to figure out where I stand with integration tests. (unless someone else gets to it first) It “feels” to like the architecture I’m proposing is simpler and more powerful, but I could be wrong.
> On Nov 7, 2017, at 7:54 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>
> I guess we are not understanding each other. If the testing language is
> AS or JS, test authors have to know how to deal with the runtime
> differences. That's why Mustella uses MXML. Automated test code
> generation could also abstract those differences from the test authors.
>
> If you want to build up a test harness of tests written in AS, I would
> recommend starting with FlexUnit (as it appears you are doing) and limit
> tests to being small units that don't require the runtime.
>
> If you want to run tests that require the runtime, I think Mustella might
> be a good starting point instead of trying to re-invent it.
>
> Of course, I could be wrong...
> -Alex
>
> On 11/7/17, 9:40 AM, "Harbs" <ha...@gmail.com> wrote:
>
>> Right. I’m proposing a totally different architecture.
>>
>> In the architecture I’m proposing, the runner is a passive observer. The
>> tests would be run by *the beads themselves* and *push* the results out
>> to the runner.
>>
>> The runner would have a count of the number of tests that are supposed to
>> be run, and when all the tests return (or a fail-early test comes back)
>> the runner exits with the pass/fail result.
>>
>> To be clear, there would be *two* separate architectures.
>>
>> 1. Unit tests would be reserved for simple tests which could be run
>> without waiting for UI things to happen. That would use an active test
>> runner.
>> 2. Integration tests would allow for complex and async tests where the
>> test runner would be passive.
>>
>> Hope this is clearer…
>> Harbs
>>
>>> On Nov 7, 2017, at 7:33 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>>>
>>> If the runner calls testBead.test(), the next line of code cannot check
>>> for results.
>>>
>>> for (i = 0; i < numTests; i++) {
>>> testBead[i].test():
>>> if (testBead[i].failed) {
>>> // record failure
>>> }
>>> }
>>
>
Re: Test Beads (was Re: Unit Tests et. al.)
Posted by Alex Harui <ah...@adobe.com.INVALID>.
I guess we are not understanding each other. If the testing language is
AS or JS, test authors have to know how to deal with the runtime
differences. That's why Mustella uses MXML. Automated test code
generation could also abstract those differences from the test authors.
If you want to build up a test harness of tests written in AS, I would
recommend starting with FlexUnit (as it appears you are doing) and limit
tests to being small units that don't require the runtime.
If you want to run tests that require the runtime, I think Mustella might
be a good starting point instead of trying to re-invent it.
Of course, I could be wrong...
-Alex
On 11/7/17, 9:40 AM, "Harbs" <ha...@gmail.com> wrote:
>Right. I’m proposing a totally different architecture.
>
>In the architecture I’m proposing, the runner is a passive observer. The
>tests would be run by *the beads themselves* and *push* the results out
>to the runner.
>
>The runner would have a count of the number of tests that are supposed to
>be run, and when all the tests return (or a fail-early test comes back)
>the runner exits with the pass/fail result.
>
>To be clear, there would be *two* separate architectures.
>
>1. Unit tests would be reserved for simple tests which could be run
>without waiting for UI things to happen. That would use an active test
>runner.
>2. Integration tests would allow for complex and async tests where the
>test runner would be passive.
>
>Hope this is clearer…
>Harbs
>
>> On Nov 7, 2017, at 7:33 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>>
>> If the runner calls testBead.test(), the next line of code cannot check
>> for results.
>>
>> for (i = 0; i < numTests; i++) {
>> testBead[i].test():
>> if (testBead[i].failed) {
>> // record failure
>> }
>> }
>
Re: Test Beads (was Re: Unit Tests et. al.)
Posted by Harbs <ha...@gmail.com>.
Right. I’m proposing a totally different architecture.
In the architecture I’m proposing, the runner is a passive observer. The tests would be run by *the beads themselves* and *push* the results out to the runner.
The runner would have a count of the number of tests that are supposed to be run, and when all the tests return (or a fail-early test comes back) the runner exits with the pass/fail result.
To be clear, there would be *two* separate architectures.
1. Unit tests would be reserved for simple tests which could be run without waiting for UI things to happen. That would use an active test runner.
2. Integration tests would allow for complex and async tests where the test runner would be passive.
Hope this is clearer…
Harbs
> On Nov 7, 2017, at 7:33 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>
> If the runner calls testBead.test(), the next line of code cannot check
> for results.
>
> for (i = 0; i < numTests; i++) {
> testBead[i].test():
> if (testBead[i].failed) {
> // record failure
> }
> }
Re: Test Beads (was Re: Unit Tests et. al.)
Posted by Alex Harui <ah...@adobe.com.INVALID>.
Snip...
On 11/7/17, 12:33 AM, "Harbs" <ha...@gmail.com> wrote:
>>To me the "Later" problem is about how to sequential lines of
>> ActionScript/JavaScript don't get run sequentially in order for the
>> runtime to do some processing. I don't understand how a bead can do
>>that
>> if the tests are written in a non-declarative language.
>
>Events. I’m making the assumption that all integration tests would be
>written inside a test bead. For example, layout testing could set some
>properties and then listen for layoutComplete to check that the layout
>was done correctly.
If the runner calls testBead.test(), the next line of code cannot check
for results.
for (i = 0; i < numTests; i++) {
testBead[i].test():
if (testBead[i].failed) {
// record failure
}
}
TestBead.test() cannot set up a listener for layoutComplete, because in
Flash and sometimes in the browser, the code may not fire layoutComplete
until the loop above finishes and gives the player or browser a chance to
render the screen (and thus determine the size and scroll parameters of a
TextField or Input element).
At least, that's my understanding.
-Alex
Re: Test Beads (was Re: Unit Tests et. al.)
Posted by Harbs <ha...@gmail.com>.
Comments inline.
> Each Mixin brings in one class, but that class can drag in tons of stuff.
> The key question for our users is how they want to determine what gets
> tested. From prior Adobe Flex users, they didn't want to add testing
> overhead to every MXML component, only certain ones, and often needed to
> assign it a different name than the id in order to have meaningful output.
> Especially because an id can be used in more than one place in MXML.
> Individual automation beads can be placed on each instance that you want
> tested, but that changes the source code. Having an external map that the
> Mixin uses to walk the DOM and add beads doesn't affect the source code.
This is clearly beyond my level of expertise. It would probably be helpful to get input from users who have had clear testing strategies in the past.
This is a topic that I’ll probably be able to better grok once we have framework unit and integration testing more solid.
> To me the "Later" problem is about how to sequential lines of
> ActionScript/JavaScript don't get run sequentially in order for the
> runtime to do some processing. I don't understand how a bead can do that
> if the tests are written in a non-declarative language.
Events. I’m making the assumption that all integration tests would be written inside a test bead. For example, layout testing could set some properties and then listen for layoutComplete to check that the layout was done correctly.
I’d probably make TestBeadBase have some kind of test() method to properly route the test and results to the main test runner.
>>
>> I’m not sure what you mean by this. What timeouts are you concerned by?
>
> Flash for sure won't let you run code for more than 60 seconds without
> letting the player do its thing. I thought there were timeouts for
> JavaScript in browsers, and potentially for operating systems thinking a
> process is "not responding". The runtime probably needs to be given a
> chance to do something between tests.
If the test runner is running in node, node is async, so this is not a problem.
If it’s running in a browser it can also be async using events and callbacks.
I’m not sure where there would be a tight loop that would cause problems.
> One theory of testing says that you should test boundary conditions of
> every code path as well as some intermediate values. Royale should have
> fewer "if" statements and other code path forks in the beads because we
> are trying to write PAYG code and every "if" theoretically introduces
> "just-in-case" code.
>
> So, in theory, if you could describe the boundary conditions in metadata,
> you could write a test case generator. I do not enjoy writing and
> debugging test cases so having something generate the tests would make
> life much easier for me.
The theory sounds good to me… ;-)
Harbs
Re: Test Beads (was Re: Unit Tests et. al.)
Posted by Alex Harui <ah...@adobe.com.INVALID>.
On 11/6/17, 2:06 PM, "Harbs" <ha...@gmail.com> wrote:
>Lots of points here.
>
>I’m not an expert either, but I’ll try to add my 2 cents…
>
>> My temptation would be to leverage the [Mixin] capability in the
>>compiler
>> instead of additional/different CSS. Then it is just a command-line
>> option to inject a class that gets initialized early and can do other
>> things (including bringing in additional/different CSS). However, I
>>have
>> been considering some sort of compiler option that injects beads on the
>> main application's strand.
>
>This sounds very interesting.
>
>That would sort of require a single bead attached to the application.
>It’s probably wor-able, but it makes fine grained testing a bit harder.
>
>I wonder if we could utilize Mixin tags to add beads to classes and MXML
>files using the same compiler option. That would allow dividing the app
>into “units” of testing where the developer thinks it makes sense.
Each Mixin brings in one class, but that class can drag in tons of stuff.
The key question for our users is how they want to determine what gets
tested. From prior Adobe Flex users, they didn't want to add testing
overhead to every MXML component, only certain ones, and often needed to
assign it a different name than the id in order to have meaningful output.
Especially because an id can be used in more than one place in MXML.
Individual automation beads can be placed on each instance that you want
tested, but that changes the source code. Having an external map that the
Mixin uses to walk the DOM and add beads doesn't affect the source code.
>
>
>> I believe the component/framework testing must figure out how to run the
>> next test step "later". And that's hard in AS and JS. Or else, we
>>needs
>> mocks or we restrict component tests to units that don't require any
>> runtime support. I'm not sure you can solve the "later" problem with
>> beads, but it would be great if you can.
>
>I think the later problem can be solved very nicely by beads. The bead
>could run tests at whatever point it wants. It could add an event
>listener to the strand and/or other beads to run specific tests at
>specific points.
To me the "Later" problem is about how to sequential lines of
ActionScript/JavaScript don't get run sequentially in order for the
runtime to do some processing. I don't understand how a bead can do that
if the tests are written in a non-declarative language.
>
>It keeps track of all its tests and sends notification to the test runner
>when it’s done with the results and/or sends the results as the
>individual tests are run. The total number of tests could be set
>manually, or it could be calculated automatically by [Test] metadata tags.
>
>> It also has to figure out how to
>> handle the script timeout issue as well. Once we decide on that, it
>>just
>> comes a matter of writing more tests.
>
>I’m not sure what you mean by this. What timeouts are you concerned by?
Flash for sure won't let you run code for more than 60 seconds without
letting the player do its thing. I thought there were timeouts for
JavaScript in browsers, and potentially for operating systems thinking a
process is "not responding". The runtime probably needs to be given a
chance to do something between tests.
>
>> Since we are brainstorming, I want to mention that I have dreams of
>> automatically generating tests from metadata.
>
>Sounds like an interesting idea, but to be honest you lost me from the
>start… ;-)
>
>I think these kinds of things are fundamentally incompatible with my
>brain, and I’ll probably have a hard time wrapping my head around this…
>;-)
One theory of testing says that you should test boundary conditions of
every code path as well as some intermediate values. Royale should have
fewer "if" statements and other code path forks in the beads because we
are trying to write PAYG code and every "if" theoretically introduces
"just-in-case" code.
So, in theory, if you could describe the boundary conditions in metadata,
you could write a test case generator. I do not enjoy writing and
debugging test cases so having something generate the tests would make
life much easier for me.
My 2 cents,
-Alex
>
>> On Nov 6, 2017, at 8:35 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>>
>> Disclaimer: I am not an expert on automated testing, but I was involved
>> in many discussions around the time Flex was donated to Apache. So I
>>have
>> some knowledge, but it might be stale. Here are some thoughts on this
>> topic.
>>
>> To respond to the subject: as in the skinning/theming thread, I
>>wouldn't
>> worry about beads right now. Beads are just encapsulations of code
>> snippets. In complex situations like these, it is often better just to
>> "get the code to work", then get someone else to "get the code to work"
>>in
>> a different scenario and then see what needs to be parameterized and
>> re-used.
>>
>> I'm unclear as to how much we need to do along the lines of automated
>> testing for Applications. There are existing tools tuned for automating
>> Application testing. It would be great to hear from users as to whether
>> they have already chosen an automated testing tool for other
>>Applications.
>> Flex, for example, provided integration with the QTP testing system.
>> Maybe people want us to leverage QTP or RIATest, or something else.
>>Also,
>> Microsoft was trying to formalize automated testing for Windows app. I
>> don't know if our users are using that or not.
>>
>> Microsoft was introducing the notion of "roles" as part of the WAI-ARIA
>> standard [1] and building a test harness around that. We've spent a
>> little bit of time thinking about that in Royale. The NumericStepper is
>> no longer a single component like it was in Flex, but rather, two
>> components (Input and up/down control) in order to conform to WAI-ARIA
>>not
>> just for testing but someday for accessibility.
>>
>> Because of beads, there should be relatively few "private" parts to a
>> component, so I don't know how much code will be needed to access
>>things,
>> especially in JS where nothing is truly private anyway.
>>
>> Because of PAYG, we do want to have some other code set the additional
>> information the automated testing tools need. IIRC, not every tag in
>>MXML
>> needs to be tested, so adding a bead to specific MXML tags to mark them
>> for the testing tools makes sense, but then you can't make it completely
>> go away at runtime.
>>
>> I often thought a key feature of PAYG and automated testing would be
>>that,
>> without touching the code, you could add some compiler option and inject
>> all of the extra data. I think this is technically possible, and I
>>think
>> this is what you are discussing in this thread, but I'm not sure if
>>folks
>> want that or not. If you don't want to touch the code, managing an
>> external map instead might be too painful. Don't know, we should just
>>try
>> it.
>>
>> My temptation would be to leverage the [Mixin] capability in the
>>compiler
>> instead of additional/different CSS. Then it is just a command-line
>> option to inject a class that gets initialized early and can do other
>> things (including bringing in additional/different CSS). However, I
>>have
>> been considering some sort of compiler option that injects beads on the
>> main application's strand.
>>
>> But the above is all about automated Application testing. IMO,
>> component/framework testing is different.
>>
>> I believe the component/framework testing must figure out how to run the
>> next test step "later". And that's hard in AS and JS. Or else, we
>>needs
>> mocks or we restrict component tests to units that don't require any
>> runtime support. I'm not sure you can solve the "later" problem with
>> beads, but it would be great if you can. It also has to figure out how
>>to
>> handle the script timeout issue as well. Once we decide on that, it
>>just
>> comes a matter of writing more tests.
>>
>> Since we are brainstorming, I want to mention that I have dreams of
>> automatically generating tests from metadata. Our framework code has
>>very
>> few functions/methods that are called by the Application developer.
>> Instead, most of the code we write are functions as setters and getters,
>> and event handlers. Adding metadata to each of our functions seems way
>> more efficient than writing tests for each one, and might help solve the
>> "later" problem as the test harness could have control over when to make
>> the function call and when to test for the results.
>>
>> So, some getter could have metadata that is something like:
>>
>> [Test[type="getter", initialValue="0", minValue="int.MIN_VALUE",
>> maxValue="int.MAX_VALUE")]
>> Function get value():int;
>>
>> And that would generate several tests:
>>
>> var comp:Foo = new Foo();
>> Assert(comp.value, is(0))
>>
>> comp.foo = int.MIN_VALUE;
>> Assert(comp.value, is(int.MIN_VALUE));
>>
>> comp.foo = int.MAX_VALUE;
>> Assert(comp.value, is(int.MAX_VALUE));
>>
>> And even, if we add more metadata about out-of-range:
>>
>> [Test[initialValue="0", minValue="0", maxValue="int.MAX_VALUE",
>> outOfRangeMin="exception")]
>> Function get value():int;
>>
>> try {
>> comp.foo = -1; // (minValue - 1)
>> } catch (e:Error) {
>> Success();
>> }
>> Failure();
>>
>> [Test[initialValue="0", minValue="0", maxValue="int.MAX_VALUE",
>> outOfRangeMin="0")]
>> Function get value():int;
>>
>> comp.foo = -1; // (minValue - 1)
>> Assert(comp.value, is(0))
>>
>>
>>
>> An Event handler might look like:
>>
>> [Test("eventType='org.apache.flex.events.MouseEvent', type="click",
>> data="localx:0;localy:0" resultEvent="stateChange")]
>> function clickHandler(e:MouseEvent):void
>> {
>> }
>>
>>
>>
>>
>>
>>
>> And result in:
>> var comp:Foo = new Foo();
>> Var e:Event = new org.apache.flex.events.MouseEvent('click');
>> e["localx"] = 0;
>> e["localy"] = 0;
>> Comp.addEventListener("stateChange", genericEventListener);
>> comp.clickHandler(e);
>> AssertEvent(was(0))
>>
>>
>> If we want to do integration testing that requires the runtime, we could
>> add a "wait" tag to the metadata and the test engine would do what it
>> needs to in order for the runtime to do some processing.
>>
>> My 2 cents,
>> -Alex
>>
>> [1]
>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.w3.o
>>rg%2FWAI%2Fintro%2Faria&data=02%7C01%7C%7Cefd3cb6bdd3d419c71bd08d52562bb4
>>9%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636456028337090213&sdata=d
>>xS8C4NfVbqXRmymt1sbJxQ322csY2hVEKZnV%2Fvvwuw%3D&reserved=0
>>
>> On 11/5/17, 1:14 AM, "Harbs" <ha...@gmail.com> wrote:
>>
>>> I wanted to branch this into a separate discussion because I want to
>>> discuss whether this is a good idea or a bad idea on its own.
>>>
>>> Harbs
>>>> On Nov 5, 2017, at 11:55 AM, Harbs <ha...@gmail.com> wrote:
>>>>
>>>> I just had an interesting idea for solving the component testing
>>>> problem in a Royale-specific way which might be a nice advantage over
>>>> other frameworks:
>>>>
>>>> Testing Beads.
>>>>
>>>> The problem with component test seem to be the following:
>>>> 1. Testing at the correct point in the component lifecycle.
>>>> 2. Being able to address specific components and their parts.
>>>> 3. Being able to fail-early on tests that don’t require complete
>>>> loading.
>>>> 4. Ensuring that all tests complete — which usually means synchronous
>>>> execution of tests.
>>>>
>>>> Testing beads seem like they should be able to solve these problems in
>>>> an interesting way.
>>>>
>>>> Basically, a testing bead would be a bead which has an interface
>>>>which:
>>>> a. Reports test passes.
>>>> b. reports test failures.
>>>> c. reports ignored test.
>>>> d. Reports when all tests are done.
>>>>
>>>> It would work something like this:
>>>> 1. A test runner/test app, would create components and add testing
>>>> beads to the components.
>>>> 2. It would retain references to the testing beads and listen for
>>>> results from the beads.
>>>> 3. The test runner would run the app.
>>>> 4. Each test bead would take care of running its own tests and report
>>>> back when done.
>>>> 5. Once all the test beads report success or a bead reports failure,
>>>> the test runner would exit with the full report.
>>>>
>>>> This would have the following advantages:
>>>> 1. All tests could run in parallel. This would probably speed up test
>>>> runs tremendously. Async operations would not block other tests from
>>>> being run.
>>>> 2. There’s no need for the test runner to worry about life-cycles. The
>>>> bead would be responsible to test at the correct point in the
>>>>lifecycle.
>>>> 3. The first test to fail could exit. Failing early could make the
>>>>test
>>>> run much quicker when tests fail.
>>>> 4. You could have an option to have the test runner either report all
>>>> failing tests or fail early on the first one.
>>>> 5. Running tests should be simple with a well-defined interface, and
>>>> the actual tests could be as simple or as complicated as necessary.
>>>>
>>>> This seems like a very good solution from framework development.
>>>>
>>>> I’m not sure how this concept could be used for application
>>>> development. I guess an application developer could create a parallel
>>>> testing app which is the same as the app plus testing beads, but that
>>>> seems awkward.
>>>>
>>>> Maybe it’s possible to use a testing CSS file which would add testing
>>>> beads to components for testing builds, the problem with that is that
>>>> there’s a requirement for code to add those beads.
>>>>
>>>> Maybe we can add special tags for adding the beads via MXML and/or
>>>> ActionScript which could be stripped out for non-test builds.
>>>>
>>>> Food for thought…
>>>> Harbs
>>>
>>
>
Re: Test Beads (was Re: Unit Tests et. al.)
Posted by Harbs <ha...@gmail.com>.
Lots of points here.
I’m not an expert either, but I’ll try to add my 2 cents…
> My temptation would be to leverage the [Mixin] capability in the compiler
> instead of additional/different CSS. Then it is just a command-line
> option to inject a class that gets initialized early and can do other
> things (including bringing in additional/different CSS). However, I have
> been considering some sort of compiler option that injects beads on the
> main application's strand.
This sounds very interesting.
That would sort of require a single bead attached to the application. It’s probably wor-able, but it makes fine grained testing a bit harder.
I wonder if we could utilize Mixin tags to add beads to classes and MXML files using the same compiler option. That would allow dividing the app into “units” of testing where the developer thinks it makes sense.
> I believe the component/framework testing must figure out how to run the
> next test step "later". And that's hard in AS and JS. Or else, we needs
> mocks or we restrict component tests to units that don't require any
> runtime support. I'm not sure you can solve the "later" problem with
> beads, but it would be great if you can.
I think the later problem can be solved very nicely by beads. The bead could run tests at whatever point it wants. It could add an event listener to the strand and/or other beads to run specific tests at specific points.
It keeps track of all its tests and sends notification to the test runner when it’s done with the results and/or sends the results as the individual tests are run. The total number of tests could be set manually, or it could be calculated automatically by [Test] metadata tags.
> It also has to figure out how to
> handle the script timeout issue as well. Once we decide on that, it just
> comes a matter of writing more tests.
I’m not sure what you mean by this. What timeouts are you concerned by?
> Since we are brainstorming, I want to mention that I have dreams of
> automatically generating tests from metadata.
Sounds like an interesting idea, but to be honest you lost me from the start… ;-)
I think these kinds of things are fundamentally incompatible with my brain, and I’ll probably have a hard time wrapping my head around this… ;-)
> On Nov 6, 2017, at 8:35 PM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>
> Disclaimer: I am not an expert on automated testing, but I was involved
> in many discussions around the time Flex was donated to Apache. So I have
> some knowledge, but it might be stale. Here are some thoughts on this
> topic.
>
> To respond to the subject: as in the skinning/theming thread, I wouldn't
> worry about beads right now. Beads are just encapsulations of code
> snippets. In complex situations like these, it is often better just to
> "get the code to work", then get someone else to "get the code to work" in
> a different scenario and then see what needs to be parameterized and
> re-used.
>
> I'm unclear as to how much we need to do along the lines of automated
> testing for Applications. There are existing tools tuned for automating
> Application testing. It would be great to hear from users as to whether
> they have already chosen an automated testing tool for other Applications.
> Flex, for example, provided integration with the QTP testing system.
> Maybe people want us to leverage QTP or RIATest, or something else. Also,
> Microsoft was trying to formalize automated testing for Windows app. I
> don't know if our users are using that or not.
>
> Microsoft was introducing the notion of "roles" as part of the WAI-ARIA
> standard [1] and building a test harness around that. We've spent a
> little bit of time thinking about that in Royale. The NumericStepper is
> no longer a single component like it was in Flex, but rather, two
> components (Input and up/down control) in order to conform to WAI-ARIA not
> just for testing but someday for accessibility.
>
> Because of beads, there should be relatively few "private" parts to a
> component, so I don't know how much code will be needed to access things,
> especially in JS where nothing is truly private anyway.
>
> Because of PAYG, we do want to have some other code set the additional
> information the automated testing tools need. IIRC, not every tag in MXML
> needs to be tested, so adding a bead to specific MXML tags to mark them
> for the testing tools makes sense, but then you can't make it completely
> go away at runtime.
>
> I often thought a key feature of PAYG and automated testing would be that,
> without touching the code, you could add some compiler option and inject
> all of the extra data. I think this is technically possible, and I think
> this is what you are discussing in this thread, but I'm not sure if folks
> want that or not. If you don't want to touch the code, managing an
> external map instead might be too painful. Don't know, we should just try
> it.
>
> My temptation would be to leverage the [Mixin] capability in the compiler
> instead of additional/different CSS. Then it is just a command-line
> option to inject a class that gets initialized early and can do other
> things (including bringing in additional/different CSS). However, I have
> been considering some sort of compiler option that injects beads on the
> main application's strand.
>
> But the above is all about automated Application testing. IMO,
> component/framework testing is different.
>
> I believe the component/framework testing must figure out how to run the
> next test step "later". And that's hard in AS and JS. Or else, we needs
> mocks or we restrict component tests to units that don't require any
> runtime support. I'm not sure you can solve the "later" problem with
> beads, but it would be great if you can. It also has to figure out how to
> handle the script timeout issue as well. Once we decide on that, it just
> comes a matter of writing more tests.
>
> Since we are brainstorming, I want to mention that I have dreams of
> automatically generating tests from metadata. Our framework code has very
> few functions/methods that are called by the Application developer.
> Instead, most of the code we write are functions as setters and getters,
> and event handlers. Adding metadata to each of our functions seems way
> more efficient than writing tests for each one, and might help solve the
> "later" problem as the test harness could have control over when to make
> the function call and when to test for the results.
>
> So, some getter could have metadata that is something like:
>
> [Test[type="getter", initialValue="0", minValue="int.MIN_VALUE",
> maxValue="int.MAX_VALUE")]
> Function get value():int;
>
> And that would generate several tests:
>
> var comp:Foo = new Foo();
> Assert(comp.value, is(0))
>
> comp.foo = int.MIN_VALUE;
> Assert(comp.value, is(int.MIN_VALUE));
>
> comp.foo = int.MAX_VALUE;
> Assert(comp.value, is(int.MAX_VALUE));
>
> And even, if we add more metadata about out-of-range:
>
> [Test[initialValue="0", minValue="0", maxValue="int.MAX_VALUE",
> outOfRangeMin="exception")]
> Function get value():int;
>
> try {
> comp.foo = -1; // (minValue - 1)
> } catch (e:Error) {
> Success();
> }
> Failure();
>
> [Test[initialValue="0", minValue="0", maxValue="int.MAX_VALUE",
> outOfRangeMin="0")]
> Function get value():int;
>
> comp.foo = -1; // (minValue - 1)
> Assert(comp.value, is(0))
>
>
>
> An Event handler might look like:
>
> [Test("eventType='org.apache.flex.events.MouseEvent', type="click",
> data="localx:0;localy:0" resultEvent="stateChange")]
> function clickHandler(e:MouseEvent):void
> {
> }
>
>
>
>
>
>
> And result in:
> var comp:Foo = new Foo();
> Var e:Event = new org.apache.flex.events.MouseEvent('click');
> e["localx"] = 0;
> e["localy"] = 0;
> Comp.addEventListener("stateChange", genericEventListener);
> comp.clickHandler(e);
> AssertEvent(was(0))
>
>
> If we want to do integration testing that requires the runtime, we could
> add a "wait" tag to the metadata and the test engine would do what it
> needs to in order for the runtime to do some processing.
>
> My 2 cents,
> -Alex
>
> [1] https://www.w3.org/WAI/intro/aria
>
> On 11/5/17, 1:14 AM, "Harbs" <ha...@gmail.com> wrote:
>
>> I wanted to branch this into a separate discussion because I want to
>> discuss whether this is a good idea or a bad idea on its own.
>>
>> Harbs
>>> On Nov 5, 2017, at 11:55 AM, Harbs <ha...@gmail.com> wrote:
>>>
>>> I just had an interesting idea for solving the component testing
>>> problem in a Royale-specific way which might be a nice advantage over
>>> other frameworks:
>>>
>>> Testing Beads.
>>>
>>> The problem with component test seem to be the following:
>>> 1. Testing at the correct point in the component lifecycle.
>>> 2. Being able to address specific components and their parts.
>>> 3. Being able to fail-early on tests that don’t require complete
>>> loading.
>>> 4. Ensuring that all tests complete — which usually means synchronous
>>> execution of tests.
>>>
>>> Testing beads seem like they should be able to solve these problems in
>>> an interesting way.
>>>
>>> Basically, a testing bead would be a bead which has an interface which:
>>> a. Reports test passes.
>>> b. reports test failures.
>>> c. reports ignored test.
>>> d. Reports when all tests are done.
>>>
>>> It would work something like this:
>>> 1. A test runner/test app, would create components and add testing
>>> beads to the components.
>>> 2. It would retain references to the testing beads and listen for
>>> results from the beads.
>>> 3. The test runner would run the app.
>>> 4. Each test bead would take care of running its own tests and report
>>> back when done.
>>> 5. Once all the test beads report success or a bead reports failure,
>>> the test runner would exit with the full report.
>>>
>>> This would have the following advantages:
>>> 1. All tests could run in parallel. This would probably speed up test
>>> runs tremendously. Async operations would not block other tests from
>>> being run.
>>> 2. There’s no need for the test runner to worry about life-cycles. The
>>> bead would be responsible to test at the correct point in the lifecycle.
>>> 3. The first test to fail could exit. Failing early could make the test
>>> run much quicker when tests fail.
>>> 4. You could have an option to have the test runner either report all
>>> failing tests or fail early on the first one.
>>> 5. Running tests should be simple with a well-defined interface, and
>>> the actual tests could be as simple or as complicated as necessary.
>>>
>>> This seems like a very good solution from framework development.
>>>
>>> I’m not sure how this concept could be used for application
>>> development. I guess an application developer could create a parallel
>>> testing app which is the same as the app plus testing beads, but that
>>> seems awkward.
>>>
>>> Maybe it’s possible to use a testing CSS file which would add testing
>>> beads to components for testing builds, the problem with that is that
>>> there’s a requirement for code to add those beads.
>>>
>>> Maybe we can add special tags for adding the beads via MXML and/or
>>> ActionScript which could be stripped out for non-test builds.
>>>
>>> Food for thought…
>>> Harbs
>>
>
Re: Test Beads (was Re: Unit Tests et. al.)
Posted by Alex Harui <ah...@adobe.com.INVALID>.
Disclaimer: I am not an expert on automated testing, but I was involved
in many discussions around the time Flex was donated to Apache. So I have
some knowledge, but it might be stale. Here are some thoughts on this
topic.
To respond to the subject: as in the skinning/theming thread, I wouldn't
worry about beads right now. Beads are just encapsulations of code
snippets. In complex situations like these, it is often better just to
"get the code to work", then get someone else to "get the code to work" in
a different scenario and then see what needs to be parameterized and
re-used.
I'm unclear as to how much we need to do along the lines of automated
testing for Applications. There are existing tools tuned for automating
Application testing. It would be great to hear from users as to whether
they have already chosen an automated testing tool for other Applications.
Flex, for example, provided integration with the QTP testing system.
Maybe people want us to leverage QTP or RIATest, or something else. Also,
Microsoft was trying to formalize automated testing for Windows app. I
don't know if our users are using that or not.
Microsoft was introducing the notion of "roles" as part of the WAI-ARIA
standard [1] and building a test harness around that. We've spent a
little bit of time thinking about that in Royale. The NumericStepper is
no longer a single component like it was in Flex, but rather, two
components (Input and up/down control) in order to conform to WAI-ARIA not
just for testing but someday for accessibility.
Because of beads, there should be relatively few "private" parts to a
component, so I don't know how much code will be needed to access things,
especially in JS where nothing is truly private anyway.
Because of PAYG, we do want to have some other code set the additional
information the automated testing tools need. IIRC, not every tag in MXML
needs to be tested, so adding a bead to specific MXML tags to mark them
for the testing tools makes sense, but then you can't make it completely
go away at runtime.
I often thought a key feature of PAYG and automated testing would be that,
without touching the code, you could add some compiler option and inject
all of the extra data. I think this is technically possible, and I think
this is what you are discussing in this thread, but I'm not sure if folks
want that or not. If you don't want to touch the code, managing an
external map instead might be too painful. Don't know, we should just try
it.
My temptation would be to leverage the [Mixin] capability in the compiler
instead of additional/different CSS. Then it is just a command-line
option to inject a class that gets initialized early and can do other
things (including bringing in additional/different CSS). However, I have
been considering some sort of compiler option that injects beads on the
main application's strand.
But the above is all about automated Application testing. IMO,
component/framework testing is different.
I believe the component/framework testing must figure out how to run the
next test step "later". And that's hard in AS and JS. Or else, we needs
mocks or we restrict component tests to units that don't require any
runtime support. I'm not sure you can solve the "later" problem with
beads, but it would be great if you can. It also has to figure out how to
handle the script timeout issue as well. Once we decide on that, it just
comes a matter of writing more tests.
Since we are brainstorming, I want to mention that I have dreams of
automatically generating tests from metadata. Our framework code has very
few functions/methods that are called by the Application developer.
Instead, most of the code we write are functions as setters and getters,
and event handlers. Adding metadata to each of our functions seems way
more efficient than writing tests for each one, and might help solve the
"later" problem as the test harness could have control over when to make
the function call and when to test for the results.
So, some getter could have metadata that is something like:
[Test[type="getter", initialValue="0", minValue="int.MIN_VALUE",
maxValue="int.MAX_VALUE")]
Function get value():int;
And that would generate several tests:
var comp:Foo = new Foo();
Assert(comp.value, is(0))
comp.foo = int.MIN_VALUE;
Assert(comp.value, is(int.MIN_VALUE));
comp.foo = int.MAX_VALUE;
Assert(comp.value, is(int.MAX_VALUE));
And even, if we add more metadata about out-of-range:
[Test[initialValue="0", minValue="0", maxValue="int.MAX_VALUE",
outOfRangeMin="exception")]
Function get value():int;
try {
comp.foo = -1; // (minValue - 1)
} catch (e:Error) {
Success();
}
Failure();
[Test[initialValue="0", minValue="0", maxValue="int.MAX_VALUE",
outOfRangeMin="0")]
Function get value():int;
comp.foo = -1; // (minValue - 1)
Assert(comp.value, is(0))
An Event handler might look like:
[Test("eventType='org.apache.flex.events.MouseEvent', type="click",
data="localx:0;localy:0" resultEvent="stateChange")]
function clickHandler(e:MouseEvent):void
{
}
And result in:
var comp:Foo = new Foo();
Var e:Event = new org.apache.flex.events.MouseEvent('click');
e["localx"] = 0;
e["localy"] = 0;
Comp.addEventListener("stateChange", genericEventListener);
comp.clickHandler(e);
AssertEvent(was(0))
If we want to do integration testing that requires the runtime, we could
add a "wait" tag to the metadata and the test engine would do what it
needs to in order for the runtime to do some processing.
My 2 cents,
-Alex
[1] https://www.w3.org/WAI/intro/aria
On 11/5/17, 1:14 AM, "Harbs" <ha...@gmail.com> wrote:
>I wanted to branch this into a separate discussion because I want to
>discuss whether this is a good idea or a bad idea on its own.
>
>Harbs
>> On Nov 5, 2017, at 11:55 AM, Harbs <ha...@gmail.com> wrote:
>>
>> I just had an interesting idea for solving the component testing
>>problem in a Royale-specific way which might be a nice advantage over
>>other frameworks:
>>
>> Testing Beads.
>>
>> The problem with component test seem to be the following:
>> 1. Testing at the correct point in the component lifecycle.
>> 2. Being able to address specific components and their parts.
>> 3. Being able to fail-early on tests that don’t require complete
>>loading.
>> 4. Ensuring that all tests complete — which usually means synchronous
>>execution of tests.
>>
>> Testing beads seem like they should be able to solve these problems in
>>an interesting way.
>>
>> Basically, a testing bead would be a bead which has an interface which:
>> a. Reports test passes.
>> b. reports test failures.
>> c. reports ignored test.
>> d. Reports when all tests are done.
>>
>> It would work something like this:
>> 1. A test runner/test app, would create components and add testing
>>beads to the components.
>> 2. It would retain references to the testing beads and listen for
>>results from the beads.
>> 3. The test runner would run the app.
>> 4. Each test bead would take care of running its own tests and report
>>back when done.
>> 5. Once all the test beads report success or a bead reports failure,
>>the test runner would exit with the full report.
>>
>> This would have the following advantages:
>> 1. All tests could run in parallel. This would probably speed up test
>>runs tremendously. Async operations would not block other tests from
>>being run.
>> 2. There’s no need for the test runner to worry about life-cycles. The
>>bead would be responsible to test at the correct point in the lifecycle.
>> 3. The first test to fail could exit. Failing early could make the test
>>run much quicker when tests fail.
>> 4. You could have an option to have the test runner either report all
>>failing tests or fail early on the first one.
>> 5. Running tests should be simple with a well-defined interface, and
>>the actual tests could be as simple or as complicated as necessary.
>>
>> This seems like a very good solution from framework development.
>>
>> I’m not sure how this concept could be used for application
>>development. I guess an application developer could create a parallel
>>testing app which is the same as the app plus testing beads, but that
>>seems awkward.
>>
>> Maybe it’s possible to use a testing CSS file which would add testing
>>beads to components for testing builds, the problem with that is that
>>there’s a requirement for code to add those beads.
>>
>> Maybe we can add special tags for adding the beads via MXML and/or
>>ActionScript which could be stripped out for non-test builds.
>>
>> Food for thought…
>> Harbs
>
Test Beads (was Re: Unit Tests et. al.)
Posted by Harbs <ha...@gmail.com>.
I wanted to branch this into a separate discussion because I want to discuss whether this is a good idea or a bad idea on its own.
Harbs
> On Nov 5, 2017, at 11:55 AM, Harbs <ha...@gmail.com> wrote:
>
> I just had an interesting idea for solving the component testing problem in a Royale-specific way which might be a nice advantage over other frameworks:
>
> Testing Beads.
>
> The problem with component test seem to be the following:
> 1. Testing at the correct point in the component lifecycle.
> 2. Being able to address specific components and their parts.
> 3. Being able to fail-early on tests that don’t require complete loading.
> 4. Ensuring that all tests complete — which usually means synchronous execution of tests.
>
> Testing beads seem like they should be able to solve these problems in an interesting way.
>
> Basically, a testing bead would be a bead which has an interface which:
> a. Reports test passes.
> b. reports test failures.
> c. reports ignored test.
> d. Reports when all tests are done.
>
> It would work something like this:
> 1. A test runner/test app, would create components and add testing beads to the components.
> 2. It would retain references to the testing beads and listen for results from the beads.
> 3. The test runner would run the app.
> 4. Each test bead would take care of running its own tests and report back when done.
> 5. Once all the test beads report success or a bead reports failure, the test runner would exit with the full report.
>
> This would have the following advantages:
> 1. All tests could run in parallel. This would probably speed up test runs tremendously. Async operations would not block other tests from being run.
> 2. There’s no need for the test runner to worry about life-cycles. The bead would be responsible to test at the correct point in the lifecycle.
> 3. The first test to fail could exit. Failing early could make the test run much quicker when tests fail.
> 4. You could have an option to have the test runner either report all failing tests or fail early on the first one.
> 5. Running tests should be simple with a well-defined interface, and the actual tests could be as simple or as complicated as necessary.
>
> This seems like a very good solution from framework development.
>
> I’m not sure how this concept could be used for application development. I guess an application developer could create a parallel testing app which is the same as the app plus testing beads, but that seems awkward.
>
> Maybe it’s possible to use a testing CSS file which would add testing beads to components for testing builds, the problem with that is that there’s a requirement for code to add those beads.
>
> Maybe we can add special tags for adding the beads via MXML and/or ActionScript which could be stripped out for non-test builds.
>
> Food for thought…
> Harbs
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
On these two points:
> Hmm, that would be cool if we don't need Selenium to report results.
> Keeping Selenium synced up is a PITA.
For tests that are not browser-dependant, we should not need selenium. Testing directly through node should be much faster besides requiring less setup.
For tests that rely on browser features, most of the testing frameworks I’ve seen use selenium. Most drive it using node.js. Some use PhantomJS, and I think there’s an option to run Chromium headlessly. One interesting framework which does not seem to use Selenium is cypress.
I’m still looking around. If anyone has experience with JS testing frameworks, please let me know...
>
> I've never looked to see how FlexUnit handles this, but I'm not
> clear how a test can be written in AS as:
>
> @Test
> Function MyTest() {
> SetAProperty();
> AssertSomeOtherProperty();
> }
I’m not sure how FlexUnit does it either.
Here are two interesting options
http://nightwatchjs.org/ <http://nightwatchjs.org/>
https://www.cypress.io/ <https://www.cypress.io/>
The frameworks I’ve seen seem to have a “wait” function.
We could probably also use event listeners.
Here’s a nice list:
https://www.joecolantonio.com/2016/06/14/top-8-essential-javascript-automation-frameworks/ <https://www.joecolantonio.com/2016/06/14/top-8-essential-javascript-automation-frameworks/>
I just had an interesting idea for solving the component testing problem in a Royale-specific way which might be a nice advantage over other frameworks:
Testing Beads.
The problem with component test seem to be the following:
1. Testing at the correct point in the component lifecycle.
2. Being able to address specific components and their parts.
3. Being able to fail-early on tests that don’t require complete loading.
4. Ensuring that all tests complete — which usually means synchronous execution of tests.
Testing beads seem like they should be able to solve these problems in an interesting way.
Basically, a testing bead would be a bead which has an interface which:
a. Reports test passes.
b. reports test failures.
c. reports ignored test.
d. Reports when all tests are done.
It would work something like this:
1. A test runner/test app, would create components and add testing beads to the components.
2. It would retain references to the testing beads and listen for results from the beads.
3. The test runner would run the app.
4. Each test bead would take care of running its own tests and report back when done.
5. Once all the test beads report success or a bead reports failure, the test runner would exit with the full report.
This would have the following advantages:
1. All tests could run in parallel. This would probably speed up test runs tremendously. Async operations would not block other tests from being run.
2. There’s no need for the test runner to worry about life-cycles. The bead would be responsible to test at the correct point in the lifecycle.
3. The first test to fail could exit. Failing early could make the test run much quicker when tests fail.
4. You could have an option to have the test runner either report all failing tests or fail early on the first one.
5. Running tests should be simple with a well-defined interface, and the actual tests could be as simple or as complicated as necessary.
This seems like a very good solution from framework development.
I’m not sure how this concept could be used for application development. I guess an application developer could create a parallel testing app which is the same as the app plus testing beads, but that seems awkward.
Maybe it’s possible to use a testing CSS file which would add testing beads to components for testing builds, the problem with that is that there’s a requirement for code to add those beads.
Maybe we can add special tags for adding the beads via MXML and/or ActionScript which could be stripped out for non-test builds.
Food for thought…
Harbs
Re: Unit Tests et. al.
Posted by Alex Harui <ah...@adobe.com.INVALID>.
Hmm, that would be cool if we don't need Selenium to report results.
Keeping Selenium synced up is a PITA.
Regarding declarative languages, the reason I chose MXML when writing
Mustella was that, in Flash, you simply have to stop running AS code in
order for the player to render the screen and determine the final size of
things. I've never looked to see how FlexUnit handles this, but I'm not
clear how a test can be written in AS as:
@Test
Function MyTest() {
SetAProperty();
AssertSomeOtherProperty();
}
unless there are mocks for the player, or you don't run any tests that
depend on the player.
In Mustella, the same test looks like:
<TestCase id="MyTest">
<SetProperty ... />
<AssertPropertyValue .../>
</TestCase>
And then on Flash, the Mustella engine sets the property and waits for the
player to refresh before running AssertProperty. And in the browser, the
Mustella engine doesn't have to wait. But at Adobe, the "Unit" was a
component, not APIs or methods within a component, and thus we needed the
player to do its thing in most cases. FlexUnit is working against smaller
units and Mustella is really more of a integration testing engine.
My other point was that, since the set of available test steps is
well-defined, you can't write a test with infinite loops, or screw up the
setting of a property, etc.
HTH,
-Alex
On 11/4/17, 10:46 PM, "Harbs" <ha...@gmail.com> wrote:
>Josh got his tests to be reported in the terminal console and he added
>integration with Travis. Integration of node processes is pretty standard
>in the JS world. I have not yet looked into integration with ant (or
>Maven), but I can’t imagine that running node via ant is much different
>than any other process.
>
>> The advantage of declarative testing languages is that you get to
>> constrain the kinds of things that a test can do, and abstract when the
>> test harness does it. It is much harder to interrupt sequential lines
>>of
>> ActionScript/JavaScript.
>
>I did not get this at all. Do you have any examples or links which show
>what you mean?
>
>I’m somewhat embarrassed to say that I’ve never really used much
>automated testing, so a lot of these concepts are more theoretical to me
>than they really should be… ;-)
>
>Harbs
>
>> On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>>
>> Those warnings do not seem to affect the success of the tests on SWF for
>> me.
>>
>> IMO, the big challenge is in capturing test results so it can be
>>reported
>> in a CI dashboard. Having a GUI display results in Flash or in a
>>browser
>> is a great first step, but I believe going from there to getting
>>Selenium
>> to report individual test failures may be trickier.
>>
>> Peter tried to write up Mustella a long time ago. It might need
>>updating.
>> See:
>>
>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.ap
>>ache.org%2Fconfluence%2Fdisplay%2FFLEX%2FMustella%2BOverview&data=02%7C01
>>%7C%7C0414ee815f254be8230f08d524190130%7Cfa7b1b5a7b34438794aed2c178decee1
>>%7C0%7C0%7C636454612170971092&sdata=70%2BMRX6UcjeFP%2BB43zSZDo53bgq0oxOZT
>>4G6uLtYzII%3D&reserved=0
>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.a
>>pache.org%2Fconfluence%2Fdisplay%2FFLEX%2FMustella%2BOverview&data=02%7C0
>>1%7C%7C0414ee815f254be8230f08d524190130%7Cfa7b1b5a7b34438794aed2c178decee
>>1%7C0%7C0%7C636454612170971092&sdata=70%2BMRX6UcjeFP%2BB43zSZDo53bgq0oxOZ
>>T4G6uLtYzII%3D&reserved=0>
>>
>> The advantage of declarative testing languages is that you get to
>> constrain the kinds of things that a test can do, and abstract when the
>> test harness does it. It is much harder to interrupt sequential lines
>>of
>> ActionScript/JavaScript.
>>
>> HTH,
>> -Alex
>>
>> On 11/4/17, 3:13 PM, "Harbs" <harbs.lists@gmail.com
>><ma...@gmail.com>> wrote:
>>
>>> It looks like FlexUnit is currently broken.
>>>
>>> I’m getting the following output when trying to run the FlexUnit tests
>>>on
>>> Core and Basic:
>>>
>>> [mxmlc]
>>>
>>>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexu
>>>ni
>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in
>>>the
>>> SWC
>>>
>>>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexu
>>>ni
>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>> [mxmlc]
>>> [mxmlc]
>>> [mxmlc]
>>>
>>>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexu
>>>ni
>>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>>> mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper
>>>in
>>> the SWC
>>>
>>>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexu
>>>ni
>>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>>
>>> Has anyone looked at this yet?
>>>
>>> Harbs
>>>
>>>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
>>>>
>>>> Thanks! That looks very useful.
>>>>
>>>> I started work on a feature/testing branch. I’ll try to merge what you
>>>> did into what Josh did. I’m going to try to get the existing Core
>>>>and/or
>>>> Basic tests working in both swf and node js output tomorrow. We’ll see
>>>> how well I do… ;-)
>>>>
>>>> We might need different configs for different testing environments,
>>>>but
>>>> I’ll see if we can keep the divergence of the environs is minimal as
>>>> possible.
>>>>
>>>> Once I get the basics worked out, I’ll likely have an idea what others
>>>> can help with. Let’s see if we can whip the Royale tests into shape.
>>>>:-)
>>>>
>>>> Harbs
>>>>
>>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
>>>>>
>>>>> Harbs, I don't know if it still works since changes to Royale, but I
>>>>> had
>>>>> something rudimentary for cross-target unit testing quite shortly
>>>>>after
>>>>> working on the reflection support last year, in fact that was my
>>>>> primary
>>>>> reason for choosing to focus on reflection at the time.
>>>>>
>>>>> It was visual output only (i.e. just to look at the test results
>>>>> output in
>>>>> the browser) and the goal was to get some compatibility with flexunit
>>>>> (so
>>>>> that the same swf-based flexunit tests that were running in the build
>>>>> -e..g
>>>>> BinaryData tests- could be run visually side by side between swf and
>>>>> js).
>>>>> I also added a new TestVariance metadata to account for known
>>>>> (expected?)
>>>>> variation between swf and js. This was needed to cover (for example)
>>>>> things
>>>>> like testing Reflection classes themselves, where the results can be
>>>>> different between the targets based on the 'native' base classes
>>>>>(e..g
>>>>> EventDispatcher inheritance chain).
>>>>>
>>>>> Some of that code might also be useful to harvest for what you are
>>>>> doing,
>>>>> I'm not sure...
>>>>>
>>>>> It's here:
>>>>>
>>>>>
>>>>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
>>>>>b
>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgith
>>>>>ub>.
>>>>>
>>>>>com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%2Fmanualtests%2FUnitTests&
>>>>>da
>>>>>
>>>>>ta=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794a
>>>>>ed
>>>>>
>>>>>2c178decee1%7C0%7C0%7C636454304479378270&sdata=DX9gXiLqUoFQizp6xrakUwx
>>>>>jl
>>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <harbs.lists@gmail.com
>>>>><ma...@gmail.com>> wrote:
>>>>>
>>>>>> Awesome!
>>>>>>
>>>>>> I assume that this means I can code the code to the Royale repo
>>>>>>under
>>>>>> Apache 2. Right?
>>>>>>
>>>>>> Harbs
>>>>>>
>>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <joshtynjala@apache.org
>>>>>>><ma...@apache.org>>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Please feel free to use my test runner. I'm happy to commit it
>>>>>>> somewhere
>>>>>> to make it official Apache code, if necessary.
>>>>>>>
>>>>>>> - Josh
>>>>>>>
>>>>>>> On 2017-11-03 04:19, Harbs <harbs.lists@gmail.com
>>>>>>><ma...@gmail.com>> wrote:
>>>>>>>> One topic which keeps coming up is better test coverage for
>>>>>>>>Royale.
>>>>>>>>
>>>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good
>>>>>>>>test
>>>>>> coverage for confidence of quality and confidence that we don’t
>>>>>> introduce
>>>>>> recessive bugs.
>>>>>>>> 2. It’s really hard to accept Github pull requests without
>>>>>>>> examining
>>>>>> the code VERY well that it does not introduce recessive bugs. CI
>>>>>> which runs
>>>>>> automated tests could give a preliminary test on pull requests to
>>>>>> ensure
>>>>>> that they don’t break anything. If the pull requests do break
>>>>>> something,
>>>>>> it allows the requester to fix the problem with confidence without
>>>>>> taking
>>>>>> others’ time.
>>>>>>>>
>>>>>>>> I think we need to break up testing into pieces and figure out a
>>>>>> strategy to implement automated tests in a way that they are
>>>>>> maintainable.
>>>>>>>>
>>>>>>>> Some points:
>>>>>>>> 1. I think integration into something like Travis would be very
>>>>>>>> helpful.
>>>>>>>> 2. I think there’s a Jenkins plugin for building pull requests.
>>>>>>>> Not
>>>>>> sure how useful it is.[1]
>>>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
>>>>>>>> which
>>>>>> could be useful for unit tests on parts of the framework which
>>>>>> don’t rely
>>>>>> on browser features. (i.e. models and the like) He mentioned the
>>>>>> possibility of donating it, and I think that would be a very useful
>>>>>> feature.
>>>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>>>> integrations using Selenium.[2][3]
>>>>>>>> 5. I think the main testing effort needs to be using JS and
>>>>>>>> something
>>>>>> like Josh’s testing framework for non-UI pieces and some
>>>>>>easy-to-use
>>>>>> Selenium integration for visual pieces and integration tests.
>>>>>>>> 6. We probably also want some API endpoints we can test off of for
>>>>>> integration testing.
>>>>>>>>
>>>>>>>> I’m willing to invest time into this.
>>>>>>>>
>>>>>>>> It’s going to be a lot of work building out the tests and I
>>>>>>>>think
>>>>>>>> we
>>>>>> need a plan for that. My thoughts:
>>>>>>>>
>>>>>>>> 1. Step one is to make it easy to write meaningful automated tests
>>>>>>>> and
>>>>>> establish a clear pattern.
>>>>>>>> 2. Step two is to start writing tests starting from the
>>>>>> most-used/easiest to beak pieces and work out from there.
>>>>>>>> 3. Once the pattern is established, any new pieces MUST have
>>>>>>>>testing
>>>>>> coverage.
>>>>>>>> 4. When fixing bugs, attention should be paid to adding testing
>>>>>>>>for
>>>>>> that component.
>>>>>>>> 5. When a pull request comes in on a piece which does not have
>>>>>>>>unit
>>>>>> test, a test must be written before accepting the pull request. The
>>>>>> test
>>>>>> does not need to be written by the requester. Before examining the
>>>>>> request,
>>>>>> the test should be written to pass for expected behavior and fail
>>>>>>for
>>>>>> the
>>>>>> bug that the pull request is attempting to fix (assuming the pull
>>>>>> request
>>>>>> is to fix a bug).
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>> Harbs
>>>>>>>>
>>>>>>>> P.S. I’m thinking of coming to the US in late December/early
>>>>>>>> January.
>>>>>> I would be interested in getting together for a hacking session with
>>>>>> folks
>>>>>> who are available.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>[1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>>>Fw
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>>>>>>>>>
>>>>>>>>
>>>>>>>>iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&data=02%7C01%7C%
>>>>>>>>7C
>>>>>>>>
>>>>>>>>67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1
>>>>>>>>%7
>>>>>>>>
>>>>>>>>C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M0uS
>>>>>>>>l3
>>>>>>>> f0ezbSZY%3D&reserved=0+
>>>>>> request+builder+plugin
>>>>>>
>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwik
>>>>>>i
>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwik
>>>>>>i>.
>>>>>>
>>>>>>jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa
>>>>>>7b
>>>>>>
>>>>>>1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=eqe6e
>>>>>>c%
>>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
>>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>>>>
>>>>>>>>
>>>>>>>>[2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>>>Fd
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fd
>>>>>>>>>
>>>>>>>> ocs.travis-ci.com
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Foc
>>>>>>>>s.travis-ci.com%2F&data=02%7C01%7C%7C0414ee815f254be8230f08d5241901
>>>>>>>>30%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454612170971092&
>>>>>>>>sdata=pnhMpAYNngULtMbB%2BHh43o8kTA5%2Bxu9sOciWDDW3dLM%3D&reserved=0
>>>>>>>>>%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%
>>>>>>>>
>>>>>>>>7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178d
>>>>>>>>ec
>>>>>>>>
>>>>>>>>ee1%7C0%7C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeO
>>>>>>>>hw
>>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
>>>>>>
>>>>>>
>>>>>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs
>>>>>>.t
>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdoc
>>>>>>s.t>
>>>>>> ravis-ci.com
>>>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fravi
>>>>>>s-ci.com%2F&data=02%7C01%7C%7C0414ee815f254be8230f08d524190130%7Cfa7b
>>>>>>1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454612170971092&sdata=jOUFN
>>>>>>%2FhuadJ3C%2B7U6xfEvhq5EZpT9OfFWaMCAkfhg%2FY%3D&reserved=0>%2Fuser%2F
>>>>>>gui-and-headless-browsers%2F&data=02%7C01%7C%7C67
>>>>>>
>>>>>>a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0
>>>>>>%7
>>>>>>
>>>>>>C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03sjOytut
>>>>>>B4
>>>>>> rGCY%3D&reserved=0>
>>>>>>>>
>>>>>>>>
>>>>>>>>[3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2
>>>>>>>>Fs
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs
>>>>>>>>>
>>>>>>>> aucelabs.com
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fau
>>>>>>>>celabs.com%2F&data=02%7C01%7C%7C0414ee815f254be8230f08d524190130%7C
>>>>>>>>fa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454612170971092&sdata
>>>>>>>>=yfrl9UGpLIpcUZRLUqyAiPxp64tMr5ow7aC8ZqhQFOA%3D&reserved=0>%2Fprodu
>>>>>>>>cts%2Fintegrations&data=02%7C01%7C%7C67a58a332b69
>>>>>>>>
>>>>>>>>42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C6
>>>>>>>>36
>>>>>>>>
>>>>>>>>454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%2FNXSRVOQho9qUW1mqdw
>>>>>>>>%3
>>>>>>>> D&reserved=0
>>>>>>>>
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs
>>>>>>>>au
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs
>>>>>>>>au>
>>>>>>>> celabs.com
>>>>>>>><https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fce
>>>>>>>>labs.com%2F&data=02%7C01%7C%7C0414ee815f254be8230f08d524190130%7Cfa
>>>>>>>>7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454612170971092&sdata=A
>>>>>>>>Nmmle6KQFo99Z3l8ie1q9xIe4SJOMQK%2FgCQbYipRwo%3D&reserved=0>%2F&data
>>>>>>>>=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa
>>>>>>>>
>>>>>>>>7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=x
>>>>>>>>pG
>>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
>>>>>> products/integrations>
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
Josh got his tests to be reported in the terminal console and he added integration with Travis. Integration of node processes is pretty standard in the JS world. I have not yet looked into integration with ant (or Maven), but I can’t imagine that running node via ant is much different than any other process.
> The advantage of declarative testing languages is that you get to
> constrain the kinds of things that a test can do, and abstract when the
> test harness does it. It is much harder to interrupt sequential lines of
> ActionScript/JavaScript.
I did not get this at all. Do you have any examples or links which show what you mean?
I’m somewhat embarrassed to say that I’ve never really used much automated testing, so a lot of these concepts are more theoretical to me than they really should be… ;-)
Harbs
> On Nov 5, 2017, at 8:17 AM, Alex Harui <ah...@adobe.com.INVALID> wrote:
>
> Those warnings do not seem to affect the success of the tests on SWF for
> me.
>
> IMO, the big challenge is in capturing test results so it can be reported
> in a CI dashboard. Having a GUI display results in Flash or in a browser
> is a great first step, but I believe going from there to getting Selenium
> to report individual test failures may be trickier.
>
> Peter tried to write up Mustella a long time ago. It might need updating.
> See:
> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview <https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview>
>
> The advantage of declarative testing languages is that you get to
> constrain the kinds of things that a test can do, and abstract when the
> test harness does it. It is much harder to interrupt sequential lines of
> ActionScript/JavaScript.
>
> HTH,
> -Alex
>
> On 11/4/17, 3:13 PM, "Harbs" <harbs.lists@gmail.com <ma...@gmail.com>> wrote:
>
>> It looks like FlexUnit is currently broken.
>>
>> I’m getting the following output when trying to run the FlexUnit tests on
>> Core and Basic:
>>
>> [mxmlc]
>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>> mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in the
>> SWC
>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>> [mxmlc]
>> [mxmlc]
>> [mxmlc]
>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>> t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>> mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper in
>> the SWC
>> /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>> t-4.3.0-20140410-as3_4.12.0.swc could not be found
>>
>> Has anyone looked at this yet?
>>
>> Harbs
>>
>>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
>>>
>>> Thanks! That looks very useful.
>>>
>>> I started work on a feature/testing branch. I’ll try to merge what you
>>> did into what Josh did. I’m going to try to get the existing Core and/or
>>> Basic tests working in both swf and node js output tomorrow. We’ll see
>>> how well I do… ;-)
>>>
>>> We might need different configs for different testing environments, but
>>> I’ll see if we can keep the divergence of the environs is minimal as
>>> possible.
>>>
>>> Once I get the basics worked out, I’ll likely have an idea what others
>>> can help with. Let’s see if we can whip the Royale tests into shape. :-)
>>>
>>> Harbs
>>>
>>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
>>>>
>>>> Harbs, I don't know if it still works since changes to Royale, but I
>>>> had
>>>> something rudimentary for cross-target unit testing quite shortly after
>>>> working on the reflection support last year, in fact that was my
>>>> primary
>>>> reason for choosing to focus on reflection at the time.
>>>>
>>>> It was visual output only (i.e. just to look at the test results
>>>> output in
>>>> the browser) and the goal was to get some compatibility with flexunit
>>>> (so
>>>> that the same swf-based flexunit tests that were running in the build
>>>> -e..g
>>>> BinaryData tests- could be run visually side by side between swf and
>>>> js).
>>>> I also added a new TestVariance metadata to account for known
>>>> (expected?)
>>>> variation between swf and js. This was needed to cover (for example)
>>>> things
>>>> like testing Reflection classes themselves, where the results can be
>>>> different between the targets based on the 'native' base classes (e..g
>>>> EventDispatcher inheritance chain).
>>>>
>>>> Some of that code might also be useful to harvest for what you are
>>>> doing,
>>>> I'm not sure...
>>>>
>>>> It's here:
>>>>
>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub>.
>>>> com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%2Fmanualtests%2FUnitTests&da
>>>> ta=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed
>>>> 2c178decee1%7C0%7C0%7C636454304479378270&sdata=DX9gXiLqUoFQizp6xrakUwxjl
>>>> cdyHMlZO5mOzDSiKDQ%3D&reserved=0
>>>>
>>>>
>>>>
>>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <harbs.lists@gmail.com <ma...@gmail.com>> wrote:
>>>>
>>>>> Awesome!
>>>>>
>>>>> I assume that this means I can code the code to the Royale repo under
>>>>> Apache 2. Right?
>>>>>
>>>>> Harbs
>>>>>
>>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <joshtynjala@apache.org <ma...@apache.org>>
>>>>>> wrote:
>>>>>>
>>>>>> Please feel free to use my test runner. I'm happy to commit it
>>>>>> somewhere
>>>>> to make it official Apache code, if necessary.
>>>>>>
>>>>>> - Josh
>>>>>>
>>>>>> On 2017-11-03 04:19, Harbs <harbs.lists@gmail.com <ma...@gmail.com>> wrote:
>>>>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>>>>
>>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>>>> coverage for confidence of quality and confidence that we don’t
>>>>> introduce
>>>>> recessive bugs.
>>>>>>> 2. It’s really hard to accept Github pull requests without
>>>>>>> examining
>>>>> the code VERY well that it does not introduce recessive bugs. CI
>>>>> which runs
>>>>> automated tests could give a preliminary test on pull requests to
>>>>> ensure
>>>>> that they don’t break anything. If the pull requests do break
>>>>> something,
>>>>> it allows the requester to fix the problem with confidence without
>>>>> taking
>>>>> others’ time.
>>>>>>>
>>>>>>> I think we need to break up testing into pieces and figure out a
>>>>> strategy to implement automated tests in a way that they are
>>>>> maintainable.
>>>>>>>
>>>>>>> Some points:
>>>>>>> 1. I think integration into something like Travis would be very
>>>>>>> helpful.
>>>>>>> 2. I think there’s a Jenkins plugin for building pull requests.
>>>>>>> Not
>>>>> sure how useful it is.[1]
>>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
>>>>>>> which
>>>>> could be useful for unit tests on parts of the framework which
>>>>> don’t rely
>>>>> on browser features. (i.e. models and the like) He mentioned the
>>>>> possibility of donating it, and I think that would be a very useful
>>>>> feature.
>>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>>> integrations using Selenium.[2][3]
>>>>>>> 5. I think the main testing effort needs to be using JS and
>>>>>>> something
>>>>> like Josh’s testing framework for non-UI pieces and some easy-to-use
>>>>> Selenium integration for visual pieces and integration tests.
>>>>>>> 6. We probably also want some API endpoints we can test off of for
>>>>> integration testing.
>>>>>>>
>>>>>>> I’m willing to invest time into this.
>>>>>>>
>>>>>>> It’s going to be a lot of work building out the tests and I think
>>>>>>> we
>>>>> need a plan for that. My thoughts:
>>>>>>>
>>>>>>> 1. Step one is to make it easy to write meaningful automated tests
>>>>>>> and
>>>>> establish a clear pattern.
>>>>>>> 2. Step two is to start writing tests starting from the
>>>>> most-used/easiest to beak pieces and work out from there.
>>>>>>> 3. Once the pattern is established, any new pieces MUST have testing
>>>>> coverage.
>>>>>>> 4. When fixing bugs, attention should be paid to adding testing for
>>>>> that component.
>>>>>>> 5. When a pull request comes in on a piece which does not have unit
>>>>> test, a test must be written before accepting the pull request. The
>>>>> test
>>>>> does not need to be written by the requester. Before examining the
>>>>> request,
>>>>> the test should be written to pass for expected behavior and fail for
>>>>> the
>>>>> bug that the pull request is attempting to fix (assuming the pull
>>>>> request
>>>>> is to fix a bug).
>>>>>>>
>>>>>>> Thoughts?
>>>>>>> Harbs
>>>>>>>
>>>>>>> P.S. I’m thinking of coming to the US in late December/early
>>>>>>> January.
>>>>> I would be interested in getting together for a hacking session with
>>>>> folks
>>>>> who are available.
>>>>>>>
>>>>>>>
>>>>>>> [1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw>
>>>>>>> iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&data=02%7C01%7C%7C
>>>>>>> 67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7
>>>>>>> C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M0uSl3
>>>>>>> f0ezbSZY%3D&reserved=0+
>>>>> request+builder+plugin
>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki>.
>>>>> jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b
>>>>> 1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=eqe6ec%
>>>>> 2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>>>
>>>>>>> [2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fd <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fd>
>>>>>>> ocs.travis-ci.com <http://ocs.travis-ci.com/>%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%
>>>>>>> 7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178dec
>>>>>>> ee1%7C0%7C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw
>>>>>>> 03sjOytutB4rGCY%3D&reserved=0 <
>>>>>
>>>>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.t <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.t>
>>>>> ravis-ci.com <http://ravis-ci.com/>%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C67
>>>>> a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7
>>>>> C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03sjOytutB4
>>>>> rGCY%3D&reserved=0>
>>>>>>>
>>>>>>> [3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs>
>>>>>>> aucelabs.com <http://aucelabs.com/>%2Fproducts%2Fintegrations&data=02%7C01%7C%7C67a58a332b69
>>>>>>> 42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636
>>>>>>> 454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%2FNXSRVOQho9qUW1mqdw%3
>>>>>>> D&reserved=0
>>>>>>> <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsau <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsau>
>>>>>>> celabs.com <http://celabs.com/>%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa
>>>>>>> 7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=xpG
>>>>>>> G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
>>>>> products/integrations>
Re: Unit Tests et. al.
Posted by Alex Harui <ah...@adobe.com.INVALID>.
Those warnings do not seem to affect the success of the tests on SWF for
me.
IMO, the big challenge is in capturing test results so it can be reported
in a CI dashboard. Having a GUI display results in Flash or in a browser
is a great first step, but I believe going from there to getting Selenium
to report individual test failures may be trickier.
Peter tried to write up Mustella a long time ago. It might need updating.
See:
https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
The advantage of declarative testing languages is that you get to
constrain the kinds of things that a test can do, and abstract when the
test harness does it. It is much harder to interrupt sequential lines of
ActionScript/JavaScript.
HTH,
-Alex
On 11/4/17, 3:13 PM, "Harbs" <ha...@gmail.com> wrote:
>It looks like FlexUnit is currently broken.
>
>I’m getting the following output when trying to run the FlexUnit tests on
>Core and Basic:
>
> [mxmlc]
>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in the
>SWC
>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>t-4.3.0-20140410-as3_4.12.0.swc could not be found
> [mxmlc]
> [mxmlc]
> [mxmlc]
>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>t-4.3.0-20140410-as3_4.12.0.swc Warning: The definition
>mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper in
>the SWC
>/Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexuni
>t-4.3.0-20140410-as3_4.12.0.swc could not be found
>
>Has anyone looked at this yet?
>
>Harbs
>
>> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
>>
>> Thanks! That looks very useful.
>>
>> I started work on a feature/testing branch. I’ll try to merge what you
>>did into what Josh did. I’m going to try to get the existing Core and/or
>>Basic tests working in both swf and node js output tomorrow. We’ll see
>>how well I do… ;-)
>>
>> We might need different configs for different testing environments, but
>>I’ll see if we can keep the divergence of the environs is minimal as
>>possible.
>>
>> Once I get the basics worked out, I’ll likely have an idea what others
>>can help with. Let’s see if we can whip the Royale tests into shape. :-)
>>
>> Harbs
>>
>>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
>>>
>>> Harbs, I don't know if it still works since changes to Royale, but I
>>>had
>>> something rudimentary for cross-target unit testing quite shortly after
>>> working on the reflection support last year, in fact that was my
>>>primary
>>> reason for choosing to focus on reflection at the time.
>>>
>>> It was visual output only (i.e. just to look at the test results
>>>output in
>>> the browser) and the goal was to get some compatibility with flexunit
>>>(so
>>> that the same swf-based flexunit tests that were running in the build
>>>-e..g
>>> BinaryData tests- could be run visually side by side between swf and
>>>js).
>>> I also added a new TestVariance metadata to account for known
>>>(expected?)
>>> variation between swf and js. This was needed to cover (for example)
>>>things
>>> like testing Reflection classes themselves, where the results can be
>>> different between the targets based on the 'native' base classes (e..g
>>> EventDispatcher inheritance chain).
>>>
>>> Some of that code might also be useful to harvest for what you are
>>>doing,
>>> I'm not sure...
>>>
>>> It's here:
>>>
>>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
>>>com%2Fapache%2Froyale-asjs%2Ftree%2Fdevelop%2Fmanualtests%2FUnitTests&da
>>>ta=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed
>>>2c178decee1%7C0%7C0%7C636454304479378270&sdata=DX9gXiLqUoFQizp6xrakUwxjl
>>>cdyHMlZO5mOzDSiKDQ%3D&reserved=0
>>>
>>>
>>>
>>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com> wrote:
>>>
>>>> Awesome!
>>>>
>>>> I assume that this means I can code the code to the Royale repo under
>>>> Apache 2. Right?
>>>>
>>>> Harbs
>>>>
>>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org>
>>>>>wrote:
>>>>>
>>>>> Please feel free to use my test runner. I'm happy to commit it
>>>>>somewhere
>>>> to make it official Apache code, if necessary.
>>>>>
>>>>> - Josh
>>>>>
>>>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>>>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>>>
>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>>> coverage for confidence of quality and confidence that we don’t
>>>>introduce
>>>> recessive bugs.
>>>>>> 2. It’s really hard to accept Github pull requests without
>>>>>>examining
>>>> the code VERY well that it does not introduce recessive bugs. CI
>>>>which runs
>>>> automated tests could give a preliminary test on pull requests to
>>>>ensure
>>>> that they don’t break anything. If the pull requests do break
>>>>something,
>>>> it allows the requester to fix the problem with confidence without
>>>>taking
>>>> others’ time.
>>>>>>
>>>>>> I think we need to break up testing into pieces and figure out a
>>>> strategy to implement automated tests in a way that they are
>>>>maintainable.
>>>>>>
>>>>>> Some points:
>>>>>> 1. I think integration into something like Travis would be very
>>>>>>helpful.
>>>>>> 2. I think there’s a Jenkins plugin for building pull requests.
>>>>>>Not
>>>> sure how useful it is.[1]
>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
>>>>>>which
>>>> could be useful for unit tests on parts of the framework which
>>>>don’t rely
>>>> on browser features. (i.e. models and the like) He mentioned the
>>>> possibility of donating it, and I think that would be a very useful
>>>>feature.
>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>> integrations using Selenium.[2][3]
>>>>>> 5. I think the main testing effort needs to be using JS and
>>>>>>something
>>>> like Josh’s testing framework for non-UI pieces and some easy-to-use
>>>> Selenium integration for visual pieces and integration tests.
>>>>>> 6. We probably also want some API endpoints we can test off of for
>>>> integration testing.
>>>>>>
>>>>>> I’m willing to invest time into this.
>>>>>>
>>>>>> It’s going to be a lot of work building out the tests and I think
>>>>>>we
>>>> need a plan for that. My thoughts:
>>>>>>
>>>>>> 1. Step one is to make it easy to write meaningful automated tests
>>>>>>and
>>>> establish a clear pattern.
>>>>>> 2. Step two is to start writing tests starting from the
>>>> most-used/easiest to beak pieces and work out from there.
>>>>>> 3. Once the pattern is established, any new pieces MUST have testing
>>>> coverage.
>>>>>> 4. When fixing bugs, attention should be paid to adding testing for
>>>> that component.
>>>>>> 5. When a pull request comes in on a piece which does not have unit
>>>> test, a test must be written before accepting the pull request. The
>>>>test
>>>> does not need to be written by the requester. Before examining the
>>>>request,
>>>> the test should be written to pass for expected behavior and fail for
>>>>the
>>>> bug that the pull request is attempting to fix (assuming the pull
>>>>request
>>>> is to fix a bug).
>>>>>>
>>>>>> Thoughts?
>>>>>> Harbs
>>>>>>
>>>>>> P.S. I’m thinking of coming to the US in late December/early
>>>>>>January.
>>>> I would be interested in getting together for a hacking session with
>>>>folks
>>>> who are available.
>>>>>>
>>>>>>
>>>>>>[1]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fw
>>>>>>iki.jenkins.io%2Fdisplay%2FJENKINS%2FGitHub%2Bpull&data=02%7C01%7C%7C
>>>>>>67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7
>>>>>>C0%7C0%7C636454304479378270&sdata=cnyBtnVycpg3Aa7Hfel3dkAlez2m7M0uSl3
>>>>>>f0ezbSZY%3D&reserved=0+
>>>> request+builder+plugin
>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwiki.
>>>>jenkins.io%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b
>>>>1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=eqe6ec%
>>>>2F%2FIatSxdUuHU0Lo8o3WxySBkdSvUNa6i9NQ%2FI%3D&reserved=0
>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>>
>>>>>>[2]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fd
>>>>>>ocs.travis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%
>>>>>>7C%7C67a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178dec
>>>>>>ee1%7C0%7C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw
>>>>>>03sjOytutB4rGCY%3D&reserved=0 <
>>>>
>>>>https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs.t
>>>>ravis-ci.com%2Fuser%2Fgui-and-headless-browsers%2F&data=02%7C01%7C%7C67
>>>>a58a332b6942a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7
>>>>C0%7C636454304479378270&sdata=zK%2FdOBmWJUsF7SylIQtMQQpAeOhw03sjOytutB4
>>>>rGCY%3D&reserved=0>
>>>>>>
>>>>>>[3]https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fs
>>>>>>aucelabs.com%2Fproducts%2Fintegrations&data=02%7C01%7C%7C67a58a332b69
>>>>>>42a2022108d523d15d35%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636
>>>>>>454304479378270&sdata=kwfto9xiP0ygeBrEqGz9ucdC%2FNXSRVOQho9qUW1mqdw%3
>>>>>>D&reserved=0
>>>>>><https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fsau
>>>>>>celabs.com%2F&data=02%7C01%7C%7C67a58a332b6942a2022108d523d15d35%7Cfa
>>>>>>7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636454304479378270&sdata=xpG
>>>>>>G85mRtcFwwG7nbRfT7oLMPm4VPeHxe4trkG1wrAw%3D&reserved=0
>>>> products/integrations>
>>>>
>>>>
>>
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
It looks like FlexUnit is currently broken.
I’m getting the following output when trying to run the FlexUnit tests on Core and Basic:
[mxmlc] /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexunit-4.3.0-20140410-as3_4.12.0.swc Warning: The definition mx.utils.ObjectUtil depended on by org.flexunit.runner.Description in the SWC /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexunit-4.3.0-20140410-as3_4.12.0.swc could not be found
[mxmlc]
[mxmlc]
[mxmlc] /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexunit-4.3.0-20140410-as3_4.12.0.swc Warning: The definition mx.utils.StringUtil depended on by flexunit.framework.AsyncTestHelper in the SWC /Users/harbs/Documents/ApacheRoyale/flex-flexunit/FlexUnit4/target/flexunit-4.3.0-20140410-as3_4.12.0.swc could not be found
Has anyone looked at this yet?
Harbs
> On Nov 4, 2017, at 10:45 PM, Harbs <ha...@gmail.com> wrote:
>
> Thanks! That looks very useful.
>
> I started work on a feature/testing branch. I’ll try to merge what you did into what Josh did. I’m going to try to get the existing Core and/or Basic tests working in both swf and node js output tomorrow. We’ll see how well I do… ;-)
>
> We might need different configs for different testing environments, but I’ll see if we can keep the divergence of the environs is minimal as possible.
>
> Once I get the basics worked out, I’ll likely have an idea what others can help with. Let’s see if we can whip the Royale tests into shape. :-)
>
> Harbs
>
>> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
>>
>> Harbs, I don't know if it still works since changes to Royale, but I had
>> something rudimentary for cross-target unit testing quite shortly after
>> working on the reflection support last year, in fact that was my primary
>> reason for choosing to focus on reflection at the time.
>>
>> It was visual output only (i.e. just to look at the test results output in
>> the browser) and the goal was to get some compatibility with flexunit (so
>> that the same swf-based flexunit tests that were running in the build -e..g
>> BinaryData tests- could be run visually side by side between swf and js).
>> I also added a new TestVariance metadata to account for known (expected?)
>> variation between swf and js. This was needed to cover (for example) things
>> like testing Reflection classes themselves, where the results can be
>> different between the targets based on the 'native' base classes (e..g
>> EventDispatcher inheritance chain).
>>
>> Some of that code might also be useful to harvest for what you are doing,
>> I'm not sure...
>>
>> It's here:
>> https://github.com/apache/royale-asjs/tree/develop/manualtests/UnitTests
>>
>>
>>
>> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com> wrote:
>>
>>> Awesome!
>>>
>>> I assume that this means I can code the code to the Royale repo under
>>> Apache 2. Right?
>>>
>>> Harbs
>>>
>>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org> wrote:
>>>>
>>>> Please feel free to use my test runner. I'm happy to commit it somewhere
>>> to make it official Apache code, if necessary.
>>>>
>>>> - Josh
>>>>
>>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>>
>>>>> I think this is becoming a critical issue for a few reasons:
>>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>> coverage for confidence of quality and confidence that we don’t introduce
>>> recessive bugs.
>>>>> 2. It’s really hard to accept Github pull requests without examining
>>> the code VERY well that it does not introduce recessive bugs. CI which runs
>>> automated tests could give a preliminary test on pull requests to ensure
>>> that they don’t break anything. If the pull requests do break something,
>>> it allows the requester to fix the problem with confidence without taking
>>> others’ time.
>>>>>
>>>>> I think we need to break up testing into pieces and figure out a
>>> strategy to implement automated tests in a way that they are maintainable.
>>>>>
>>>>> Some points:
>>>>> 1. I think integration into something like Travis would be very helpful.
>>>>> 2. I think there’s a Jenkins plugin for building pull requests. Not
>>> sure how useful it is.[1]
>>>>> 3. Josh has created a Node.js-compatible test-runner architecture which
>>> could be useful for unit tests on parts of the framework which don’t rely
>>> on browser features. (i.e. models and the like) He mentioned the
>>> possibility of donating it, and I think that would be a very useful feature.
>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>> integrations using Selenium.[2][3]
>>>>> 5. I think the main testing effort needs to be using JS and something
>>> like Josh’s testing framework for non-UI pieces and some easy-to-use
>>> Selenium integration for visual pieces and integration tests.
>>>>> 6. We probably also want some API endpoints we can test off of for
>>> integration testing.
>>>>>
>>>>> I’m willing to invest time into this.
>>>>>
>>>>> It’s going to be a lot of work building out the tests and I think we
>>> need a plan for that. My thoughts:
>>>>>
>>>>> 1. Step one is to make it easy to write meaningful automated tests and
>>> establish a clear pattern.
>>>>> 2. Step two is to start writing tests starting from the
>>> most-used/easiest to beak pieces and work out from there.
>>>>> 3. Once the pattern is established, any new pieces MUST have testing
>>> coverage.
>>>>> 4. When fixing bugs, attention should be paid to adding testing for
>>> that component.
>>>>> 5. When a pull request comes in on a piece which does not have unit
>>> test, a test must be written before accepting the pull request. The test
>>> does not need to be written by the requester. Before examining the request,
>>> the test should be written to pass for expected behavior and fail for the
>>> bug that the pull request is attempting to fix (assuming the pull request
>>> is to fix a bug).
>>>>>
>>>>> Thoughts?
>>>>> Harbs
>>>>>
>>>>> P.S. I’m thinking of coming to the US in late December/early January.
>>> I would be interested in getting together for a hacking session with folks
>>> who are available.
>>>>>
>>>>> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
>>> request+builder+plugin <https://wiki.jenkins.io/
>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
>>> https://docs.travis-ci.com/user/gui-and-headless-browsers/>
>>>>> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/
>>> products/integrations>
>>>
>>>
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
Thanks! That looks very useful.
I started work on a feature/testing branch. I’ll try to merge what you did into what Josh did. I’m going to try to get the existing Core and/or Basic tests working in both swf and node js output tomorrow. We’ll see how well I do… ;-)
We might need different configs for different testing environments, but I’ll see if we can keep the divergence of the environs is minimal as possible.
Once I get the basics worked out, I’ll likely have an idea what others can help with. Let’s see if we can whip the Royale tests into shape. :-)
Harbs
> On Nov 4, 2017, at 10:30 PM, Greg Dove <gr...@gmail.com> wrote:
>
> Harbs, I don't know if it still works since changes to Royale, but I had
> something rudimentary for cross-target unit testing quite shortly after
> working on the reflection support last year, in fact that was my primary
> reason for choosing to focus on reflection at the time.
>
> It was visual output only (i.e. just to look at the test results output in
> the browser) and the goal was to get some compatibility with flexunit (so
> that the same swf-based flexunit tests that were running in the build -e..g
> BinaryData tests- could be run visually side by side between swf and js).
> I also added a new TestVariance metadata to account for known (expected?)
> variation between swf and js. This was needed to cover (for example) things
> like testing Reflection classes themselves, where the results can be
> different between the targets based on the 'native' base classes (e..g
> EventDispatcher inheritance chain).
>
> Some of that code might also be useful to harvest for what you are doing,
> I'm not sure...
>
> It's here:
> https://github.com/apache/royale-asjs/tree/develop/manualtests/UnitTests
>
>
>
> On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com> wrote:
>
>> Awesome!
>>
>> I assume that this means I can code the code to the Royale repo under
>> Apache 2. Right?
>>
>> Harbs
>>
>>> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org> wrote:
>>>
>>> Please feel free to use my test runner. I'm happy to commit it somewhere
>> to make it official Apache code, if necessary.
>>>
>>> - Josh
>>>
>>> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>
>>>> I think this is becoming a critical issue for a few reasons:
>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>> coverage for confidence of quality and confidence that we don’t introduce
>> recessive bugs.
>>>> 2. It’s really hard to accept Github pull requests without examining
>> the code VERY well that it does not introduce recessive bugs. CI which runs
>> automated tests could give a preliminary test on pull requests to ensure
>> that they don’t break anything. If the pull requests do break something,
>> it allows the requester to fix the problem with confidence without taking
>> others’ time.
>>>>
>>>> I think we need to break up testing into pieces and figure out a
>> strategy to implement automated tests in a way that they are maintainable.
>>>>
>>>> Some points:
>>>> 1. I think integration into something like Travis would be very helpful.
>>>> 2. I think there’s a Jenkins plugin for building pull requests. Not
>> sure how useful it is.[1]
>>>> 3. Josh has created a Node.js-compatible test-runner architecture which
>> could be useful for unit tests on parts of the framework which don’t rely
>> on browser features. (i.e. models and the like) He mentioned the
>> possibility of donating it, and I think that would be a very useful feature.
>>>> 4. For UI and integration tests, there seem to be some pretty cool
>> integrations using Selenium.[2][3]
>>>> 5. I think the main testing effort needs to be using JS and something
>> like Josh’s testing framework for non-UI pieces and some easy-to-use
>> Selenium integration for visual pieces and integration tests.
>>>> 6. We probably also want some API endpoints we can test off of for
>> integration testing.
>>>>
>>>> I’m willing to invest time into this.
>>>>
>>>> It’s going to be a lot of work building out the tests and I think we
>> need a plan for that. My thoughts:
>>>>
>>>> 1. Step one is to make it easy to write meaningful automated tests and
>> establish a clear pattern.
>>>> 2. Step two is to start writing tests starting from the
>> most-used/easiest to beak pieces and work out from there.
>>>> 3. Once the pattern is established, any new pieces MUST have testing
>> coverage.
>>>> 4. When fixing bugs, attention should be paid to adding testing for
>> that component.
>>>> 5. When a pull request comes in on a piece which does not have unit
>> test, a test must be written before accepting the pull request. The test
>> does not need to be written by the requester. Before examining the request,
>> the test should be written to pass for expected behavior and fail for the
>> bug that the pull request is attempting to fix (assuming the pull request
>> is to fix a bug).
>>>>
>>>> Thoughts?
>>>> Harbs
>>>>
>>>> P.S. I’m thinking of coming to the US in late December/early January.
>> I would be interested in getting together for a hacking session with folks
>> who are available.
>>>>
>>>> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
>> request+builder+plugin <https://wiki.jenkins.io/
>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
>> https://docs.travis-ci.com/user/gui-and-headless-browsers/>
>>>> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/
>> products/integrations>
>>
>>
Re: Unit Tests et. al.
Posted by Greg Dove <gr...@gmail.com>.
Harbs, I don't know if it still works since changes to Royale, but I had
something rudimentary for cross-target unit testing quite shortly after
working on the reflection support last year, in fact that was my primary
reason for choosing to focus on reflection at the time.
It was visual output only (i.e. just to look at the test results output in
the browser) and the goal was to get some compatibility with flexunit (so
that the same swf-based flexunit tests that were running in the build -e..g
BinaryData tests- could be run visually side by side between swf and js).
I also added a new TestVariance metadata to account for known (expected?)
variation between swf and js. This was needed to cover (for example) things
like testing Reflection classes themselves, where the results can be
different between the targets based on the 'native' base classes (e..g
EventDispatcher inheritance chain).
Some of that code might also be useful to harvest for what you are doing,
I'm not sure...
It's here:
https://github.com/apache/royale-asjs/tree/develop/manualtests/UnitTests
On Sun, Nov 5, 2017 at 7:30 AM, Harbs <ha...@gmail.com> wrote:
> Awesome!
>
> I assume that this means I can code the code to the Royale repo under
> Apache 2. Right?
>
> Harbs
>
> > On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org> wrote:
> >
> > Please feel free to use my test runner. I'm happy to commit it somewhere
> to make it official Apache code, if necessary.
> >
> > - Josh
> >
> > On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
> >> One topic which keeps coming up is better test coverage for Royale.
> >>
> >> I think this is becoming a critical issue for a few reasons:
> >> 1. As we get close to version 1.0 it’s necessary to have good test
> coverage for confidence of quality and confidence that we don’t introduce
> recessive bugs.
> >> 2. It’s really hard to accept Github pull requests without examining
> the code VERY well that it does not introduce recessive bugs. CI which runs
> automated tests could give a preliminary test on pull requests to ensure
> that they don’t break anything. If the pull requests do break something,
> it allows the requester to fix the problem with confidence without taking
> others’ time.
> >>
> >> I think we need to break up testing into pieces and figure out a
> strategy to implement automated tests in a way that they are maintainable.
> >>
> >> Some points:
> >> 1. I think integration into something like Travis would be very helpful.
> >> 2. I think there’s a Jenkins plugin for building pull requests. Not
> sure how useful it is.[1]
> >> 3. Josh has created a Node.js-compatible test-runner architecture which
> could be useful for unit tests on parts of the framework which don’t rely
> on browser features. (i.e. models and the like) He mentioned the
> possibility of donating it, and I think that would be a very useful feature.
> >> 4. For UI and integration tests, there seem to be some pretty cool
> integrations using Selenium.[2][3]
> >> 5. I think the main testing effort needs to be using JS and something
> like Josh’s testing framework for non-UI pieces and some easy-to-use
> Selenium integration for visual pieces and integration tests.
> >> 6. We probably also want some API endpoints we can test off of for
> integration testing.
> >>
> >> I’m willing to invest time into this.
> >>
> >> It’s going to be a lot of work building out the tests and I think we
> need a plan for that. My thoughts:
> >>
> >> 1. Step one is to make it easy to write meaningful automated tests and
> establish a clear pattern.
> >> 2. Step two is to start writing tests starting from the
> most-used/easiest to beak pieces and work out from there.
> >> 3. Once the pattern is established, any new pieces MUST have testing
> coverage.
> >> 4. When fixing bugs, attention should be paid to adding testing for
> that component.
> >> 5. When a pull request comes in on a piece which does not have unit
> test, a test must be written before accepting the pull request. The test
> does not need to be written by the requester. Before examining the request,
> the test should be written to pass for expected behavior and fail for the
> bug that the pull request is attempting to fix (assuming the pull request
> is to fix a bug).
> >>
> >> Thoughts?
> >> Harbs
> >>
> >> P.S. I’m thinking of coming to the US in late December/early January.
> I would be interested in getting together for a hacking session with folks
> who are available.
> >>
> >> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
> request+builder+plugin <https://wiki.jenkins.io/
> display/JENKINS/GitHub+pull+request+builder+plugin>
> >> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/>
> >> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/
> products/integrations>
>
>
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
Awesome!
I assume that this means I can code the code to the Royale repo under Apache 2. Right?
Harbs
> On Nov 3, 2017, at 5:37 PM, Josh Tynjala <jo...@apache.org> wrote:
>
> Please feel free to use my test runner. I'm happy to commit it somewhere to make it official Apache code, if necessary.
>
> - Josh
>
> On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
>> One topic which keeps coming up is better test coverage for Royale.
>>
>> I think this is becoming a critical issue for a few reasons:
>> 1. As we get close to version 1.0 itâs necessary to have good test coverage for confidence of quality and confidence that we donât introduce recessive bugs.
>> 2. Itâs really hard to accept Github pull requests without examining the code VERY well that it does not introduce recessive bugs. CI which runs automated tests could give a preliminary test on pull requests to ensure that they donât break anything. If the pull requests do break something, it allows the requester to fix the problem with confidence without taking othersâ time.
>>
>> I think we need to break up testing into pieces and figure out a strategy to implement automated tests in a way that they are maintainable.
>>
>> Some points:
>> 1. I think integration into something like Travis would be very helpful.
>> 2. I think thereâs a Jenkins plugin for building pull requests. Not sure how useful it is.[1]
>> 3. Josh has created a Node.js-compatible test-runner architecture which could be useful for unit tests on parts of the framework which donât rely on browser features. (i.e. models and the like) He mentioned the possibility of donating it, and I think that would be a very useful feature.
>> 4. For UI and integration tests, there seem to be some pretty cool integrations using Selenium.[2][3]
>> 5. I think the main testing effort needs to be using JS and something like Joshâs testing framework for non-UI pieces and some easy-to-use Selenium integration for visual pieces and integration tests.
>> 6. We probably also want some API endpoints we can test off of for integration testing.
>>
>> Iâm willing to invest time into this.
>>
>> Itâs going to be a lot of work building out the tests and I think we need a plan for that. My thoughts:
>>
>> 1. Step one is to make it easy to write meaningful automated tests and establish a clear pattern.
>> 2. Step two is to start writing tests starting from the most-used/easiest to beak pieces and work out from there.
>> 3. Once the pattern is established, any new pieces MUST have testing coverage.
>> 4. When fixing bugs, attention should be paid to adding testing for that component.
>> 5. When a pull request comes in on a piece which does not have unit test, a test must be written before accepting the pull request. The test does not need to be written by the requester. Before examining the request, the test should be written to pass for expected behavior and fail for the bug that the pull request is attempting to fix (assuming the pull request is to fix a bug).
>>
>> Thoughts?
>> Harbs
>>
>> P.S. Iâm thinking of coming to the US in late December/early January. I would be interested in getting together for a hacking session with folks who are available.
>>
>> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin <https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin>
>> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>
>> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/products/integrations>
Re: Unit Tests et. al.
Posted by Josh Tynjala <jo...@apache.org>.
Please feel free to use my test runner. I'm happy to commit it somewhere to make it official Apache code, if necessary.
- Josh
On 2017-11-03 04:19, Harbs <ha...@gmail.com> wrote:
> One topic which keeps coming up is better test coverage for Royale.
>
> I think this is becoming a critical issue for a few reasons:
> 1. As we get close to version 1.0 itâs necessary to have good test coverage for confidence of quality and confidence that we donât introduce recessive bugs.
> 2. Itâs really hard to accept Github pull requests without examining the code VERY well that it does not introduce recessive bugs. CI which runs automated tests could give a preliminary test on pull requests to ensure that they donât break anything. If the pull requests do break something, it allows the requester to fix the problem with confidence without taking othersâ time.
>
> I think we need to break up testing into pieces and figure out a strategy to implement automated tests in a way that they are maintainable.
>
> Some points:
> 1. I think integration into something like Travis would be very helpful.
> 2. I think thereâs a Jenkins plugin for building pull requests. Not sure how useful it is.[1]
> 3. Josh has created a Node.js-compatible test-runner architecture which could be useful for unit tests on parts of the framework which donât rely on browser features. (i.e. models and the like) He mentioned the possibility of donating it, and I think that would be a very useful feature.
> 4. For UI and integration tests, there seem to be some pretty cool integrations using Selenium.[2][3]
> 5. I think the main testing effort needs to be using JS and something like Joshâs testing framework for non-UI pieces and some easy-to-use Selenium integration for visual pieces and integration tests.
> 6. We probably also want some API endpoints we can test off of for integration testing.
>
> Iâm willing to invest time into this.
>
> Itâs going to be a lot of work building out the tests and I think we need a plan for that. My thoughts:
>
> 1. Step one is to make it easy to write meaningful automated tests and establish a clear pattern.
> 2. Step two is to start writing tests starting from the most-used/easiest to beak pieces and work out from there.
> 3. Once the pattern is established, any new pieces MUST have testing coverage.
> 4. When fixing bugs, attention should be paid to adding testing for that component.
> 5. When a pull request comes in on a piece which does not have unit test, a test must be written before accepting the pull request. The test does not need to be written by the requester. Before examining the request, the test should be written to pass for expected behavior and fail for the bug that the pull request is attempting to fix (assuming the pull request is to fix a bug).
>
> Thoughts?
> Harbs
>
> P.S. Iâm thinking of coming to the US in late December/early January. I would be interested in getting together for a hacking session with folks who are available.
>
> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin <https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin>
> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>
> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/products/integrations>
Re: Unit Tests et. al.
Posted by Piotr Zarzycki <pi...@gmail.com>.
Cool! It look like he wrote his own TestRunner. We just need something
which gave us Asserts and Runner. If it would be adopted from our resources
good, but using Josh's is also ok I think.
Piotr
2017-11-03 14:32 GMT+01:00 Harbs <ha...@gmail.com>:
> > Executing them probably is happen through
> > the junit - maybe we could use flexunit ? etc.
>
> Josh set up a pretty good system.[1]
>
> I’m hoping we can adopt that and extend it to run selenium as well.
>
> [1]https://github.com/BowlerHatLLC/asconfigc/tree/master/test <
> https://github.com/BowlerHatLLC/asconfigc/tree/master/test>
>
> > On Nov 3, 2017, at 2:57 PM, Piotr Zarzycki <pi...@gmail.com>
> wrote:
> >
> > Don't think about Maven for now. Once you build your fist test in AS3 I
> > will probably add there Maven too. In that case Maven is just building
> jar
> > and you can execute those jar. Executing them probably is happen through
> > the junit - maybe we could use flexunit ? etc.
> >
> > Here is how I understand above example:
> > 1) Maven is building jar and helping to reach our examples probably by
> > dependency: RoyaleStore
> > 2) jar has been run along with tests which are using gecko driver [1]
> > 3) Test is executed [3] -
> > If we let's say can write tests in node.js and use selenium functions -
> > that's ok cause we can wrap up all node things in AS3.
> >
> > [1] https://github.com/mozilla/geckodriver <https://github.com/mozilla/
> geckodriver>
> > [2]
> > https://github.com/apache/royale-asjs/blob/develop/examples/examples-
> integrationtests/src/test/java/org/apache/royale/examples/royalestore/
> RoyaleStoreIT.java <https://github.com/apache/royale-asjs/blob/develop/
> examples/examples-integrationtests/src/test/java/org/apache/royale/
> examples/royalestore/RoyaleStoreIT.java>
> >
> > Piotr
> >
> >
> > 2017-11-03 13:36 GMT+01:00 Harbs <harbs.lists@gmail.com <mailto:
> harbs.lists@gmail.com>>:
> >
> >> I don’t understand how they work at all. There seems to be some Maven
> >> magic going on, and I was having trouble figuring it out that last time
> I
> >> looked.
> >>
> >>> On Nov 3, 2017, at 2:23 PM, Piotr Zarzycki <pi...@gmail.com>
> >> wrote:
> >>>
> >>> Harbs,
> >>>
> >>> I haven't used them personally, but I understand how they are working.
> I
> >>> agree that we should put our effort to have those tests written in AS3
> >>> instead of Java. We don't have to many active people among the
> committer
> >>> who know Java. Those example which we have can be just starting point -
> >> Can
> >>> it be rewritten to AS3 ?
> >>>
> >>> Piotr
> >>>
> >>> 2017-11-03 13:04 GMT+01:00 Harbs <harbs.lists@gmail.com <mailto:
> harbs.lists@gmail.com> <mailto:
> >> harbs.lists@gmail.com <ma...@gmail.com>>>:
> >>>
> >>>> Hi Piotr,
> >>>>
> >>>> Do you understand how the Selenium tests work and how to write them?
> >>>>
> >>>> It looks like the tests are written in Java. IMO, ideally, the tests
> >>>> should be written in ActionScript.
> >>>>
> >>>> Selenium has a Node.js driver and it makes sense that we should be
> >> taking
> >>>> advantage of that.
> >>>>
> >>>> In fact, we can probably have one framework which does non-UI unit
> >> testing
> >>>> using Node and UI testing using the node and selenium.
> >>>>
> >>>> Harbs
> >>>>
> >>>>> On Nov 3, 2017, at 1:31 PM, Piotr Zarzycki <
> piotrzarzycki21@gmail.com <ma...@gmail.com>>
> >>>> wrote:
> >>>>>
> >>>>> Hi Harbs,
> >>>>>
> >>>>> Fully agree what you are saying! Sometimes I have found myself
> >> difficult
> >>>>> without deeper knowledge how something is working when I check the
> >> code.
> >>>>> One thing which I can mention is that we already have Selenium
> >>>> integrated.
> >>>>> If you are going to spend some time on that I would start from the
> >> point
> >>>>> [1] and see whether we can use it.
> >>>>>
> >>>>> As for your coming to US - Wish I could be there!
> >>>>>
> >>>>> [1]
> >>>>> https://github.com/apache/royale-asjs/tree/develop/
> examples/examples- <https://github.com/apache/royale-asjs/tree/develop/
> examples/examples->
> >>>> integrationtests <https://github.com/apache/royale-asjs/tree/develop/
> <https://github.com/apache/royale-asjs/tree/develop/> <
> >> https://github.com/apache/royale-asjs/tree/develop/ <
> https://github.com/apache/royale-asjs/tree/develop/>>
> >>>> examples/examples-integrationtests>
> >>>>>
> >>>>> Piotr
> >>>>>
> >>>>>
> >>>>> 2017-11-03 12:19 GMT+01:00 Harbs <harbs.lists@gmail.com <mailto:
> harbs.lists@gmail.com> <mailto:
> >> harbs.lists@gmail.com <ma...@gmail.com>> <mailto:
> >>>> harbs.lists@gmail.com <ma...@gmail.com> <mailto:
> harbs.lists@gmail.com <ma...@gmail.com>>>>:
> >>>>>
> >>>>>> One topic which keeps coming up is better test coverage for Royale.
> >>>>>>
> >>>>>> I think this is becoming a critical issue for a few reasons:
> >>>>>> 1. As we get close to version 1.0 it’s necessary to have good test
> >>>>>> coverage for confidence of quality and confidence that we don’t
> >>>> introduce
> >>>>>> recessive bugs.
> >>>>>> 2. It’s really hard to accept Github pull requests without examining
> >> the
> >>>>>> code VERY well that it does not introduce recessive bugs. CI which
> >> runs
> >>>>>> automated tests could give a preliminary test on pull requests to
> >> ensure
> >>>>>> that they don’t break anything. If the pull requests do break
> >>>> something, it
> >>>>>> allows the requester to fix the problem with confidence without
> taking
> >>>>>> others’ time.
> >>>>>>
> >>>>>> I think we need to break up testing into pieces and figure out a
> >>>> strategy
> >>>>>> to implement automated tests in a way that they are maintainable.
> >>>>>>
> >>>>>> Some points:
> >>>>>> 1. I think integration into something like Travis would be very
> >> helpful.
> >>>>>> 2. I think there’s a Jenkins plugin for building pull requests. Not
> >> sure
> >>>>>> how useful it is.[1]
> >>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
> >> which
> >>>>>> could be useful for unit tests on parts of the framework which don’t
> >>>> rely
> >>>>>> on browser features. (i.e. models and the like) He mentioned the
> >>>>>> possibility of donating it, and I think that would be a very useful
> >>>> feature.
> >>>>>> 4. For UI and integration tests, there seem to be some pretty cool
> >>>>>> integrations using Selenium.[2][3]
> >>>>>> 5. I think the main testing effort needs to be using JS and
> something
> >>>> like
> >>>>>> Josh’s testing framework for non-UI pieces and some easy-to-use
> >> Selenium
> >>>>>> integration for visual pieces and integration tests.
> >>>>>> 6. We probably also want some API endpoints we can test off of for
> >>>>>> integration testing.
> >>>>>>
> >>>>>> I’m willing to invest time into this.
> >>>>>>
> >>>>>> It’s going to be a lot of work building out the tests and I think we
> >>>> need
> >>>>>> a plan for that. My thoughts:
> >>>>>>
> >>>>>> 1. Step one is to make it easy to write meaningful automated tests
> and
> >>>>>> establish a clear pattern.
> >>>>>> 2. Step two is to start writing tests starting from the
> >>>> most-used/easiest
> >>>>>> to beak pieces and work out from there.
> >>>>>> 3. Once the pattern is established, any new pieces MUST have testing
> >>>>>> coverage.
> >>>>>> 4. When fixing bugs, attention should be paid to adding testing for
> >> that
> >>>>>> component.
> >>>>>> 5. When a pull request comes in on a piece which does not have unit
> >>>> test,
> >>>>>> a test must be written before accepting the pull request. The test
> >> does
> >>>> not
> >>>>>> need to be written by the requester. Before examining the request,
> the
> >>>> test
> >>>>>> should be written to pass for expected behavior and fail for the bug
> >>>> that
> >>>>>> the pull request is attempting to fix (assuming the pull request is
> to
> >>>> fix
> >>>>>> a bug).
> >>>>>>
> >>>>>> Thoughts?
> >>>>>> Harbs
> >>>>>>
> >>>>>> P.S. I’m thinking of coming to the US in late December/early
> January.
> >> I
> >>>>>> would be interested in getting together for a hacking session with
> >> folks
> >>>>>> who are available.
> >>>>>>
> >>>>>> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+ <
> https://wiki.jenkins.io/display/JENKINS/GitHub+pull+> <
> >> https://wiki.jenkins.io/display/JENKINS/GitHub+pull+ <
> https://wiki.jenkins.io/display/JENKINS/GitHub+pull+>>
> >>>>>> request+builder+plugin <https://wiki.jenkins.io/ <
> https://wiki.jenkins.io/> <
> >> https://wiki.jenkins.io/ <https://wiki.jenkins.io/>> <
> >>>> https://wiki.jenkins.io/ <https://wiki.jenkins.io/> <
> https://wiki.jenkins.io/ <https://wiki.jenkins.io/>>>
> >>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
> >>>>>> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
> >> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/>> <
> >>>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
> >> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/>>> <
> >>>>>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
> >> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/>> <
> >>>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
> >> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/>>>>
> >>>>>> [3]https://saucelabs.com/products/integrations <
> https://saucelabs.com/products/integrations> <
> >> https://saucelabs.com/products/integrations <https://saucelabs.com/
> products/integrations>> <https://saucelabs.com/ <https://saucelabs.com/> <
> >> https://saucelabs.com/ <https://saucelabs.com/>>
> >>>> products/integrations> <https://saucelabs.com/ <
> https://saucelabs.com/> <https://saucelabs.com/ <https://saucelabs.com/>>
> >> <https://saucelabs.com/ <https://saucelabs.com/> <
> https://saucelabs.com/ <https://saucelabs.com/>>>
> >>>>>> products/integrations>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>>
> >>>>> Piotr Zarzycki
> >>>>>
> >>>>> mobile: +48 880 859 557
> >>>>> skype: zarzycki10
> >>>>>
> >>>>> LinkedIn: http://www.linkedin.com/piotrzarzycki <
> http://www.linkedin.com/piotrzarzycki> <
> >> http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/
> piotrzarzycki>> <
> >>>> http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/
> piotrzarzycki> <http://www.linkedin.com/ <http://www.linkedin.com/>
> >> piotrzarzycki>>
> >>>>> <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
> https://pl.linkedin.com/in/piotr-zarzycki-92a53552> <
> >> https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
> https://pl.linkedin.com/in/piotr-zarzycki-92a53552>> <
> >>>> https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
> https://pl.linkedin.com/in/piotr-zarzycki-92a53552> <
> >> https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
> https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>>>
> >>>>>
> >>>>> GitHub: https://github.com/piotrzarzycki21 <https://github.com/
> piotrzarzycki21> <https://github.com/ <https://github.com/>
> >> piotrzarzycki21> <https://github.com/ <https://github.com/> <
> https://github.com/ <https://github.com/>>
> >>>> piotrzarzycki21>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Piotr Zarzycki
> >>>
> >>> mobile: +48 880 859 557
> >>> skype: zarzycki10
> >>>
> >>> LinkedIn: http://www.linkedin.com/piotrzarzycki <
> >> http://www.linkedin.com/piotrzarzycki>
> >>> <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
> >> https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>
> >>>
> >>> GitHub: https://github.com/piotrzarzycki21 <https://github.com/
> >> piotrzarzycki21>
> >>
> >
> >
> >
> > --
> >
> > Piotr Zarzycki
> >
> > mobile: +48 880 859 557
> > skype: zarzycki10
> >
> > LinkedIn: http://www.linkedin.com/piotrzarzycki
> > <https://pl.linkedin.com/in/piotr-zarzycki-92a53552>
> >
> > GitHub: https://github.com/piotrzarzycki21
>
>
--
Piotr Zarzycki
mobile: +48 880 859 557
skype: zarzycki10
LinkedIn: http://www.linkedin.com/piotrzarzycki
<https://pl.linkedin.com/in/piotr-zarzycki-92a53552>
GitHub: https://github.com/piotrzarzycki21
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
> Executing them probably is happen through
> the junit - maybe we could use flexunit ? etc.
Josh set up a pretty good system.[1]
I’m hoping we can adopt that and extend it to run selenium as well.
[1]https://github.com/BowlerHatLLC/asconfigc/tree/master/test <https://github.com/BowlerHatLLC/asconfigc/tree/master/test>
> On Nov 3, 2017, at 2:57 PM, Piotr Zarzycki <pi...@gmail.com> wrote:
>
> Don't think about Maven for now. Once you build your fist test in AS3 I
> will probably add there Maven too. In that case Maven is just building jar
> and you can execute those jar. Executing them probably is happen through
> the junit - maybe we could use flexunit ? etc.
>
> Here is how I understand above example:
> 1) Maven is building jar and helping to reach our examples probably by
> dependency: RoyaleStore
> 2) jar has been run along with tests which are using gecko driver [1]
> 3) Test is executed [3] -
> If we let's say can write tests in node.js and use selenium functions -
> that's ok cause we can wrap up all node things in AS3.
>
> [1] https://github.com/mozilla/geckodriver <https://github.com/mozilla/geckodriver>
> [2]
> https://github.com/apache/royale-asjs/blob/develop/examples/examples-integrationtests/src/test/java/org/apache/royale/examples/royalestore/RoyaleStoreIT.java <https://github.com/apache/royale-asjs/blob/develop/examples/examples-integrationtests/src/test/java/org/apache/royale/examples/royalestore/RoyaleStoreIT.java>
>
> Piotr
>
>
> 2017-11-03 13:36 GMT+01:00 Harbs <harbs.lists@gmail.com <ma...@gmail.com>>:
>
>> I don’t understand how they work at all. There seems to be some Maven
>> magic going on, and I was having trouble figuring it out that last time I
>> looked.
>>
>>> On Nov 3, 2017, at 2:23 PM, Piotr Zarzycki <pi...@gmail.com>
>> wrote:
>>>
>>> Harbs,
>>>
>>> I haven't used them personally, but I understand how they are working. I
>>> agree that we should put our effort to have those tests written in AS3
>>> instead of Java. We don't have to many active people among the committer
>>> who know Java. Those example which we have can be just starting point -
>> Can
>>> it be rewritten to AS3 ?
>>>
>>> Piotr
>>>
>>> 2017-11-03 13:04 GMT+01:00 Harbs <harbs.lists@gmail.com <ma...@gmail.com> <mailto:
>> harbs.lists@gmail.com <ma...@gmail.com>>>:
>>>
>>>> Hi Piotr,
>>>>
>>>> Do you understand how the Selenium tests work and how to write them?
>>>>
>>>> It looks like the tests are written in Java. IMO, ideally, the tests
>>>> should be written in ActionScript.
>>>>
>>>> Selenium has a Node.js driver and it makes sense that we should be
>> taking
>>>> advantage of that.
>>>>
>>>> In fact, we can probably have one framework which does non-UI unit
>> testing
>>>> using Node and UI testing using the node and selenium.
>>>>
>>>> Harbs
>>>>
>>>>> On Nov 3, 2017, at 1:31 PM, Piotr Zarzycki <piotrzarzycki21@gmail.com <ma...@gmail.com>>
>>>> wrote:
>>>>>
>>>>> Hi Harbs,
>>>>>
>>>>> Fully agree what you are saying! Sometimes I have found myself
>> difficult
>>>>> without deeper knowledge how something is working when I check the
>> code.
>>>>> One thing which I can mention is that we already have Selenium
>>>> integrated.
>>>>> If you are going to spend some time on that I would start from the
>> point
>>>>> [1] and see whether we can use it.
>>>>>
>>>>> As for your coming to US - Wish I could be there!
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/royale-asjs/tree/develop/examples/examples- <https://github.com/apache/royale-asjs/tree/develop/examples/examples->
>>>> integrationtests <https://github.com/apache/royale-asjs/tree/develop/ <https://github.com/apache/royale-asjs/tree/develop/> <
>> https://github.com/apache/royale-asjs/tree/develop/ <https://github.com/apache/royale-asjs/tree/develop/>>
>>>> examples/examples-integrationtests>
>>>>>
>>>>> Piotr
>>>>>
>>>>>
>>>>> 2017-11-03 12:19 GMT+01:00 Harbs <harbs.lists@gmail.com <ma...@gmail.com> <mailto:
>> harbs.lists@gmail.com <ma...@gmail.com>> <mailto:
>>>> harbs.lists@gmail.com <ma...@gmail.com> <mailto:harbs.lists@gmail.com <ma...@gmail.com>>>>:
>>>>>
>>>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>>>
>>>>>> I think this is becoming a critical issue for a few reasons:
>>>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>>>>> coverage for confidence of quality and confidence that we don’t
>>>> introduce
>>>>>> recessive bugs.
>>>>>> 2. It’s really hard to accept Github pull requests without examining
>> the
>>>>>> code VERY well that it does not introduce recessive bugs. CI which
>> runs
>>>>>> automated tests could give a preliminary test on pull requests to
>> ensure
>>>>>> that they don’t break anything. If the pull requests do break
>>>> something, it
>>>>>> allows the requester to fix the problem with confidence without taking
>>>>>> others’ time.
>>>>>>
>>>>>> I think we need to break up testing into pieces and figure out a
>>>> strategy
>>>>>> to implement automated tests in a way that they are maintainable.
>>>>>>
>>>>>> Some points:
>>>>>> 1. I think integration into something like Travis would be very
>> helpful.
>>>>>> 2. I think there’s a Jenkins plugin for building pull requests. Not
>> sure
>>>>>> how useful it is.[1]
>>>>>> 3. Josh has created a Node.js-compatible test-runner architecture
>> which
>>>>>> could be useful for unit tests on parts of the framework which don’t
>>>> rely
>>>>>> on browser features. (i.e. models and the like) He mentioned the
>>>>>> possibility of donating it, and I think that would be a very useful
>>>> feature.
>>>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>>>> integrations using Selenium.[2][3]
>>>>>> 5. I think the main testing effort needs to be using JS and something
>>>> like
>>>>>> Josh’s testing framework for non-UI pieces and some easy-to-use
>> Selenium
>>>>>> integration for visual pieces and integration tests.
>>>>>> 6. We probably also want some API endpoints we can test off of for
>>>>>> integration testing.
>>>>>>
>>>>>> I’m willing to invest time into this.
>>>>>>
>>>>>> It’s going to be a lot of work building out the tests and I think we
>>>> need
>>>>>> a plan for that. My thoughts:
>>>>>>
>>>>>> 1. Step one is to make it easy to write meaningful automated tests and
>>>>>> establish a clear pattern.
>>>>>> 2. Step two is to start writing tests starting from the
>>>> most-used/easiest
>>>>>> to beak pieces and work out from there.
>>>>>> 3. Once the pattern is established, any new pieces MUST have testing
>>>>>> coverage.
>>>>>> 4. When fixing bugs, attention should be paid to adding testing for
>> that
>>>>>> component.
>>>>>> 5. When a pull request comes in on a piece which does not have unit
>>>> test,
>>>>>> a test must be written before accepting the pull request. The test
>> does
>>>> not
>>>>>> need to be written by the requester. Before examining the request, the
>>>> test
>>>>>> should be written to pass for expected behavior and fail for the bug
>>>> that
>>>>>> the pull request is attempting to fix (assuming the pull request is to
>>>> fix
>>>>>> a bug).
>>>>>>
>>>>>> Thoughts?
>>>>>> Harbs
>>>>>>
>>>>>> P.S. I’m thinking of coming to the US in late December/early January.
>> I
>>>>>> would be interested in getting together for a hacking session with
>> folks
>>>>>> who are available.
>>>>>>
>>>>>> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+ <https://wiki.jenkins.io/display/JENKINS/GitHub+pull+> <
>> https://wiki.jenkins.io/display/JENKINS/GitHub+pull+ <https://wiki.jenkins.io/display/JENKINS/GitHub+pull+>>
>>>>>> request+builder+plugin <https://wiki.jenkins.io/ <https://wiki.jenkins.io/> <
>> https://wiki.jenkins.io/ <https://wiki.jenkins.io/>> <
>>>> https://wiki.jenkins.io/ <https://wiki.jenkins.io/> <https://wiki.jenkins.io/ <https://wiki.jenkins.io/>>>
>>>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>>>> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>> <
>>>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>>> <
>>>>>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>> <
>>>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>>>>
>>>>>> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/products/integrations> <
>> https://saucelabs.com/products/integrations <https://saucelabs.com/products/integrations>> <https://saucelabs.com/ <https://saucelabs.com/> <
>> https://saucelabs.com/ <https://saucelabs.com/>>
>>>> products/integrations> <https://saucelabs.com/ <https://saucelabs.com/> <https://saucelabs.com/ <https://saucelabs.com/>>
>> <https://saucelabs.com/ <https://saucelabs.com/> <https://saucelabs.com/ <https://saucelabs.com/>>>
>>>>>> products/integrations>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Piotr Zarzycki
>>>>>
>>>>> mobile: +48 880 859 557
>>>>> skype: zarzycki10
>>>>>
>>>>> LinkedIn: http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/piotrzarzycki> <
>> http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/piotrzarzycki>> <
>>>> http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/piotrzarzycki> <http://www.linkedin.com/ <http://www.linkedin.com/>
>> piotrzarzycki>>
>>>>> <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <https://pl.linkedin.com/in/piotr-zarzycki-92a53552> <
>> https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <https://pl.linkedin.com/in/piotr-zarzycki-92a53552>> <
>>>> https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <https://pl.linkedin.com/in/piotr-zarzycki-92a53552> <
>> https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>>>
>>>>>
>>>>> GitHub: https://github.com/piotrzarzycki21 <https://github.com/piotrzarzycki21> <https://github.com/ <https://github.com/>
>> piotrzarzycki21> <https://github.com/ <https://github.com/> <https://github.com/ <https://github.com/>>
>>>> piotrzarzycki21>
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Piotr Zarzycki
>>>
>>> mobile: +48 880 859 557
>>> skype: zarzycki10
>>>
>>> LinkedIn: http://www.linkedin.com/piotrzarzycki <
>> http://www.linkedin.com/piotrzarzycki>
>>> <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
>> https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>
>>>
>>> GitHub: https://github.com/piotrzarzycki21 <https://github.com/
>> piotrzarzycki21>
>>
>
>
>
> --
>
> Piotr Zarzycki
>
> mobile: +48 880 859 557
> skype: zarzycki10
>
> LinkedIn: http://www.linkedin.com/piotrzarzycki
> <https://pl.linkedin.com/in/piotr-zarzycki-92a53552>
>
> GitHub: https://github.com/piotrzarzycki21
Re: Unit Tests et. al.
Posted by Piotr Zarzycki <pi...@gmail.com>.
Don't think about Maven for now. Once you build your fist test in AS3 I
will probably add there Maven too. In that case Maven is just building jar
and you can execute those jar. Executing them probably is happen through
the junit - maybe we could use flexunit ? etc.
Here is how I understand above example:
1) Maven is building jar and helping to reach our examples probably by
dependency: RoyaleStore
2) jar has been run along with tests which are using gecko driver [1]
3) Test is executed [3] -
If we let's say can write tests in node.js and use selenium functions -
that's ok cause we can wrap up all node things in AS3.
[1] https://github.com/mozilla/geckodriver
[2]
https://github.com/apache/royale-asjs/blob/develop/examples/examples-integrationtests/src/test/java/org/apache/royale/examples/royalestore/RoyaleStoreIT.java
Piotr
2017-11-03 13:36 GMT+01:00 Harbs <ha...@gmail.com>:
> I don’t understand how they work at all. There seems to be some Maven
> magic going on, and I was having trouble figuring it out that last time I
> looked.
>
> > On Nov 3, 2017, at 2:23 PM, Piotr Zarzycki <pi...@gmail.com>
> wrote:
> >
> > Harbs,
> >
> > I haven't used them personally, but I understand how they are working. I
> > agree that we should put our effort to have those tests written in AS3
> > instead of Java. We don't have to many active people among the committer
> > who know Java. Those example which we have can be just starting point -
> Can
> > it be rewritten to AS3 ?
> >
> > Piotr
> >
> > 2017-11-03 13:04 GMT+01:00 Harbs <harbs.lists@gmail.com <mailto:
> harbs.lists@gmail.com>>:
> >
> >> Hi Piotr,
> >>
> >> Do you understand how the Selenium tests work and how to write them?
> >>
> >> It looks like the tests are written in Java. IMO, ideally, the tests
> >> should be written in ActionScript.
> >>
> >> Selenium has a Node.js driver and it makes sense that we should be
> taking
> >> advantage of that.
> >>
> >> In fact, we can probably have one framework which does non-UI unit
> testing
> >> using Node and UI testing using the node and selenium.
> >>
> >> Harbs
> >>
> >>> On Nov 3, 2017, at 1:31 PM, Piotr Zarzycki <pi...@gmail.com>
> >> wrote:
> >>>
> >>> Hi Harbs,
> >>>
> >>> Fully agree what you are saying! Sometimes I have found myself
> difficult
> >>> without deeper knowledge how something is working when I check the
> code.
> >>> One thing which I can mention is that we already have Selenium
> >> integrated.
> >>> If you are going to spend some time on that I would start from the
> point
> >>> [1] and see whether we can use it.
> >>>
> >>> As for your coming to US - Wish I could be there!
> >>>
> >>> [1]
> >>> https://github.com/apache/royale-asjs/tree/develop/examples/examples-
> >> integrationtests <https://github.com/apache/royale-asjs/tree/develop/ <
> https://github.com/apache/royale-asjs/tree/develop/>
> >> examples/examples-integrationtests>
> >>>
> >>> Piotr
> >>>
> >>>
> >>> 2017-11-03 12:19 GMT+01:00 Harbs <harbs.lists@gmail.com <mailto:
> harbs.lists@gmail.com> <mailto:
> >> harbs.lists@gmail.com <ma...@gmail.com>>>:
> >>>
> >>>> One topic which keeps coming up is better test coverage for Royale.
> >>>>
> >>>> I think this is becoming a critical issue for a few reasons:
> >>>> 1. As we get close to version 1.0 it’s necessary to have good test
> >>>> coverage for confidence of quality and confidence that we don’t
> >> introduce
> >>>> recessive bugs.
> >>>> 2. It’s really hard to accept Github pull requests without examining
> the
> >>>> code VERY well that it does not introduce recessive bugs. CI which
> runs
> >>>> automated tests could give a preliminary test on pull requests to
> ensure
> >>>> that they don’t break anything. If the pull requests do break
> >> something, it
> >>>> allows the requester to fix the problem with confidence without taking
> >>>> others’ time.
> >>>>
> >>>> I think we need to break up testing into pieces and figure out a
> >> strategy
> >>>> to implement automated tests in a way that they are maintainable.
> >>>>
> >>>> Some points:
> >>>> 1. I think integration into something like Travis would be very
> helpful.
> >>>> 2. I think there’s a Jenkins plugin for building pull requests. Not
> sure
> >>>> how useful it is.[1]
> >>>> 3. Josh has created a Node.js-compatible test-runner architecture
> which
> >>>> could be useful for unit tests on parts of the framework which don’t
> >> rely
> >>>> on browser features. (i.e. models and the like) He mentioned the
> >>>> possibility of donating it, and I think that would be a very useful
> >> feature.
> >>>> 4. For UI and integration tests, there seem to be some pretty cool
> >>>> integrations using Selenium.[2][3]
> >>>> 5. I think the main testing effort needs to be using JS and something
> >> like
> >>>> Josh’s testing framework for non-UI pieces and some easy-to-use
> Selenium
> >>>> integration for visual pieces and integration tests.
> >>>> 6. We probably also want some API endpoints we can test off of for
> >>>> integration testing.
> >>>>
> >>>> I’m willing to invest time into this.
> >>>>
> >>>> It’s going to be a lot of work building out the tests and I think we
> >> need
> >>>> a plan for that. My thoughts:
> >>>>
> >>>> 1. Step one is to make it easy to write meaningful automated tests and
> >>>> establish a clear pattern.
> >>>> 2. Step two is to start writing tests starting from the
> >> most-used/easiest
> >>>> to beak pieces and work out from there.
> >>>> 3. Once the pattern is established, any new pieces MUST have testing
> >>>> coverage.
> >>>> 4. When fixing bugs, attention should be paid to adding testing for
> that
> >>>> component.
> >>>> 5. When a pull request comes in on a piece which does not have unit
> >> test,
> >>>> a test must be written before accepting the pull request. The test
> does
> >> not
> >>>> need to be written by the requester. Before examining the request, the
> >> test
> >>>> should be written to pass for expected behavior and fail for the bug
> >> that
> >>>> the pull request is attempting to fix (assuming the pull request is to
> >> fix
> >>>> a bug).
> >>>>
> >>>> Thoughts?
> >>>> Harbs
> >>>>
> >>>> P.S. I’m thinking of coming to the US in late December/early January.
> I
> >>>> would be interested in getting together for a hacking session with
> folks
> >>>> who are available.
> >>>>
> >>>> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+ <
> https://wiki.jenkins.io/display/JENKINS/GitHub+pull+>
> >>>> request+builder+plugin <https://wiki.jenkins.io/ <
> https://wiki.jenkins.io/> <
> >> https://wiki.jenkins.io/ <https://wiki.jenkins.io/>>
> >>>> display/JENKINS/GitHub+pull+request+builder+plugin>
> >>>> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
> >> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/>> <
> >>>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
> >> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/>>>
> >>>> [3]https://saucelabs.com/products/integrations <
> https://saucelabs.com/products/integrations> <https://saucelabs.com/ <
> https://saucelabs.com/>
> >> products/integrations> <https://saucelabs.com/ <https://saucelabs.com/>
> <https://saucelabs.com/ <https://saucelabs.com/>>
> >>>> products/integrations>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>> Piotr Zarzycki
> >>>
> >>> mobile: +48 880 859 557
> >>> skype: zarzycki10
> >>>
> >>> LinkedIn: http://www.linkedin.com/piotrzarzycki <
> http://www.linkedin.com/piotrzarzycki> <
> >> http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/
> piotrzarzycki>>
> >>> <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
> https://pl.linkedin.com/in/piotr-zarzycki-92a53552> <
> >> https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
> https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>>
> >>>
> >>> GitHub: https://github.com/piotrzarzycki21 <https://github.com/
> piotrzarzycki21> <https://github.com/ <https://github.com/>
> >> piotrzarzycki21>
> >>
> >
> >
> >
> > --
> >
> > Piotr Zarzycki
> >
> > mobile: +48 880 859 557
> > skype: zarzycki10
> >
> > LinkedIn: http://www.linkedin.com/piotrzarzycki <
> http://www.linkedin.com/piotrzarzycki>
> > <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
> https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>
> >
> > GitHub: https://github.com/piotrzarzycki21 <https://github.com/
> piotrzarzycki21>
>
--
Piotr Zarzycki
mobile: +48 880 859 557
skype: zarzycki10
LinkedIn: http://www.linkedin.com/piotrzarzycki
<https://pl.linkedin.com/in/piotr-zarzycki-92a53552>
GitHub: https://github.com/piotrzarzycki21
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
I don’t understand how they work at all. There seems to be some Maven magic going on, and I was having trouble figuring it out that last time I looked.
> On Nov 3, 2017, at 2:23 PM, Piotr Zarzycki <pi...@gmail.com> wrote:
>
> Harbs,
>
> I haven't used them personally, but I understand how they are working. I
> agree that we should put our effort to have those tests written in AS3
> instead of Java. We don't have to many active people among the committer
> who know Java. Those example which we have can be just starting point - Can
> it be rewritten to AS3 ?
>
> Piotr
>
> 2017-11-03 13:04 GMT+01:00 Harbs <harbs.lists@gmail.com <ma...@gmail.com>>:
>
>> Hi Piotr,
>>
>> Do you understand how the Selenium tests work and how to write them?
>>
>> It looks like the tests are written in Java. IMO, ideally, the tests
>> should be written in ActionScript.
>>
>> Selenium has a Node.js driver and it makes sense that we should be taking
>> advantage of that.
>>
>> In fact, we can probably have one framework which does non-UI unit testing
>> using Node and UI testing using the node and selenium.
>>
>> Harbs
>>
>>> On Nov 3, 2017, at 1:31 PM, Piotr Zarzycki <pi...@gmail.com>
>> wrote:
>>>
>>> Hi Harbs,
>>>
>>> Fully agree what you are saying! Sometimes I have found myself difficult
>>> without deeper knowledge how something is working when I check the code.
>>> One thing which I can mention is that we already have Selenium
>> integrated.
>>> If you are going to spend some time on that I would start from the point
>>> [1] and see whether we can use it.
>>>
>>> As for your coming to US - Wish I could be there!
>>>
>>> [1]
>>> https://github.com/apache/royale-asjs/tree/develop/examples/examples-
>> integrationtests <https://github.com/apache/royale-asjs/tree/develop/ <https://github.com/apache/royale-asjs/tree/develop/>
>> examples/examples-integrationtests>
>>>
>>> Piotr
>>>
>>>
>>> 2017-11-03 12:19 GMT+01:00 Harbs <harbs.lists@gmail.com <ma...@gmail.com> <mailto:
>> harbs.lists@gmail.com <ma...@gmail.com>>>:
>>>
>>>> One topic which keeps coming up is better test coverage for Royale.
>>>>
>>>> I think this is becoming a critical issue for a few reasons:
>>>> 1. As we get close to version 1.0 it’s necessary to have good test
>>>> coverage for confidence of quality and confidence that we don’t
>> introduce
>>>> recessive bugs.
>>>> 2. It’s really hard to accept Github pull requests without examining the
>>>> code VERY well that it does not introduce recessive bugs. CI which runs
>>>> automated tests could give a preliminary test on pull requests to ensure
>>>> that they don’t break anything. If the pull requests do break
>> something, it
>>>> allows the requester to fix the problem with confidence without taking
>>>> others’ time.
>>>>
>>>> I think we need to break up testing into pieces and figure out a
>> strategy
>>>> to implement automated tests in a way that they are maintainable.
>>>>
>>>> Some points:
>>>> 1. I think integration into something like Travis would be very helpful.
>>>> 2. I think there’s a Jenkins plugin for building pull requests. Not sure
>>>> how useful it is.[1]
>>>> 3. Josh has created a Node.js-compatible test-runner architecture which
>>>> could be useful for unit tests on parts of the framework which don’t
>> rely
>>>> on browser features. (i.e. models and the like) He mentioned the
>>>> possibility of donating it, and I think that would be a very useful
>> feature.
>>>> 4. For UI and integration tests, there seem to be some pretty cool
>>>> integrations using Selenium.[2][3]
>>>> 5. I think the main testing effort needs to be using JS and something
>> like
>>>> Josh’s testing framework for non-UI pieces and some easy-to-use Selenium
>>>> integration for visual pieces and integration tests.
>>>> 6. We probably also want some API endpoints we can test off of for
>>>> integration testing.
>>>>
>>>> I’m willing to invest time into this.
>>>>
>>>> It’s going to be a lot of work building out the tests and I think we
>> need
>>>> a plan for that. My thoughts:
>>>>
>>>> 1. Step one is to make it easy to write meaningful automated tests and
>>>> establish a clear pattern.
>>>> 2. Step two is to start writing tests starting from the
>> most-used/easiest
>>>> to beak pieces and work out from there.
>>>> 3. Once the pattern is established, any new pieces MUST have testing
>>>> coverage.
>>>> 4. When fixing bugs, attention should be paid to adding testing for that
>>>> component.
>>>> 5. When a pull request comes in on a piece which does not have unit
>> test,
>>>> a test must be written before accepting the pull request. The test does
>> not
>>>> need to be written by the requester. Before examining the request, the
>> test
>>>> should be written to pass for expected behavior and fail for the bug
>> that
>>>> the pull request is attempting to fix (assuming the pull request is to
>> fix
>>>> a bug).
>>>>
>>>> Thoughts?
>>>> Harbs
>>>>
>>>> P.S. I’m thinking of coming to the US in late December/early January. I
>>>> would be interested in getting together for a hacking session with folks
>>>> who are available.
>>>>
>>>> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+ <https://wiki.jenkins.io/display/JENKINS/GitHub+pull+>
>>>> request+builder+plugin <https://wiki.jenkins.io/ <https://wiki.jenkins.io/> <
>> https://wiki.jenkins.io/ <https://wiki.jenkins.io/>>
>>>> display/JENKINS/GitHub+pull+request+builder+plugin>
>>>> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>> <
>>>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>>>
>>>> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/products/integrations> <https://saucelabs.com/ <https://saucelabs.com/>
>> products/integrations> <https://saucelabs.com/ <https://saucelabs.com/> <https://saucelabs.com/ <https://saucelabs.com/>>
>>>> products/integrations>
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Piotr Zarzycki
>>>
>>> mobile: +48 880 859 557
>>> skype: zarzycki10
>>>
>>> LinkedIn: http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/piotrzarzycki> <
>> http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/piotrzarzycki>>
>>> <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <https://pl.linkedin.com/in/piotr-zarzycki-92a53552> <
>> https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>>
>>>
>>> GitHub: https://github.com/piotrzarzycki21 <https://github.com/piotrzarzycki21> <https://github.com/ <https://github.com/>
>> piotrzarzycki21>
>>
>
>
>
> --
>
> Piotr Zarzycki
>
> mobile: +48 880 859 557
> skype: zarzycki10
>
> LinkedIn: http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/piotrzarzycki>
> <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>
>
> GitHub: https://github.com/piotrzarzycki21 <https://github.com/piotrzarzycki21>
Re: Unit Tests et. al.
Posted by Piotr Zarzycki <pi...@gmail.com>.
Harbs,
I haven't used them personally, but I understand how they are working. I
agree that we should put our effort to have those tests written in AS3
instead of Java. We don't have to many active people among the committer
who know Java. Those example which we have can be just starting point - Can
it be rewritten to AS3 ?
Piotr
2017-11-03 13:04 GMT+01:00 Harbs <ha...@gmail.com>:
> Hi Piotr,
>
> Do you understand how the Selenium tests work and how to write them?
>
> It looks like the tests are written in Java. IMO, ideally, the tests
> should be written in ActionScript.
>
> Selenium has a Node.js driver and it makes sense that we should be taking
> advantage of that.
>
> In fact, we can probably have one framework which does non-UI unit testing
> using Node and UI testing using the node and selenium.
>
> Harbs
>
> > On Nov 3, 2017, at 1:31 PM, Piotr Zarzycki <pi...@gmail.com>
> wrote:
> >
> > Hi Harbs,
> >
> > Fully agree what you are saying! Sometimes I have found myself difficult
> > without deeper knowledge how something is working when I check the code.
> > One thing which I can mention is that we already have Selenium
> integrated.
> > If you are going to spend some time on that I would start from the point
> > [1] and see whether we can use it.
> >
> > As for your coming to US - Wish I could be there!
> >
> > [1]
> > https://github.com/apache/royale-asjs/tree/develop/examples/examples-
> integrationtests <https://github.com/apache/royale-asjs/tree/develop/
> examples/examples-integrationtests>
> >
> > Piotr
> >
> >
> > 2017-11-03 12:19 GMT+01:00 Harbs <harbs.lists@gmail.com <mailto:
> harbs.lists@gmail.com>>:
> >
> >> One topic which keeps coming up is better test coverage for Royale.
> >>
> >> I think this is becoming a critical issue for a few reasons:
> >> 1. As we get close to version 1.0 it’s necessary to have good test
> >> coverage for confidence of quality and confidence that we don’t
> introduce
> >> recessive bugs.
> >> 2. It’s really hard to accept Github pull requests without examining the
> >> code VERY well that it does not introduce recessive bugs. CI which runs
> >> automated tests could give a preliminary test on pull requests to ensure
> >> that they don’t break anything. If the pull requests do break
> something, it
> >> allows the requester to fix the problem with confidence without taking
> >> others’ time.
> >>
> >> I think we need to break up testing into pieces and figure out a
> strategy
> >> to implement automated tests in a way that they are maintainable.
> >>
> >> Some points:
> >> 1. I think integration into something like Travis would be very helpful.
> >> 2. I think there’s a Jenkins plugin for building pull requests. Not sure
> >> how useful it is.[1]
> >> 3. Josh has created a Node.js-compatible test-runner architecture which
> >> could be useful for unit tests on parts of the framework which don’t
> rely
> >> on browser features. (i.e. models and the like) He mentioned the
> >> possibility of donating it, and I think that would be a very useful
> feature.
> >> 4. For UI and integration tests, there seem to be some pretty cool
> >> integrations using Selenium.[2][3]
> >> 5. I think the main testing effort needs to be using JS and something
> like
> >> Josh’s testing framework for non-UI pieces and some easy-to-use Selenium
> >> integration for visual pieces and integration tests.
> >> 6. We probably also want some API endpoints we can test off of for
> >> integration testing.
> >>
> >> I’m willing to invest time into this.
> >>
> >> It’s going to be a lot of work building out the tests and I think we
> need
> >> a plan for that. My thoughts:
> >>
> >> 1. Step one is to make it easy to write meaningful automated tests and
> >> establish a clear pattern.
> >> 2. Step two is to start writing tests starting from the
> most-used/easiest
> >> to beak pieces and work out from there.
> >> 3. Once the pattern is established, any new pieces MUST have testing
> >> coverage.
> >> 4. When fixing bugs, attention should be paid to adding testing for that
> >> component.
> >> 5. When a pull request comes in on a piece which does not have unit
> test,
> >> a test must be written before accepting the pull request. The test does
> not
> >> need to be written by the requester. Before examining the request, the
> test
> >> should be written to pass for expected behavior and fail for the bug
> that
> >> the pull request is attempting to fix (assuming the pull request is to
> fix
> >> a bug).
> >>
> >> Thoughts?
> >> Harbs
> >>
> >> P.S. I’m thinking of coming to the US in late December/early January. I
> >> would be interested in getting together for a hacking session with folks
> >> who are available.
> >>
> >> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
> >> request+builder+plugin <https://wiki.jenkins.io/ <
> https://wiki.jenkins.io/>
> >> display/JENKINS/GitHub+pull+request+builder+plugin>
> >> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
> >> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/>>
> >> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/
> products/integrations> <https://saucelabs.com/ <https://saucelabs.com/>
> >> products/integrations>
> >
> >
> >
> >
> > --
> >
> > Piotr Zarzycki
> >
> > mobile: +48 880 859 557
> > skype: zarzycki10
> >
> > LinkedIn: http://www.linkedin.com/piotrzarzycki <
> http://www.linkedin.com/piotrzarzycki>
> > <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <
> https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>
> >
> > GitHub: https://github.com/piotrzarzycki21 <https://github.com/
> piotrzarzycki21>
>
--
Piotr Zarzycki
mobile: +48 880 859 557
skype: zarzycki10
LinkedIn: http://www.linkedin.com/piotrzarzycki
<https://pl.linkedin.com/in/piotr-zarzycki-92a53552>
GitHub: https://github.com/piotrzarzycki21
Re: Unit Tests et. al.
Posted by Harbs <ha...@gmail.com>.
Hi Piotr,
Do you understand how the Selenium tests work and how to write them?
It looks like the tests are written in Java. IMO, ideally, the tests should be written in ActionScript.
Selenium has a Node.js driver and it makes sense that we should be taking advantage of that.
In fact, we can probably have one framework which does non-UI unit testing using Node and UI testing using the node and selenium.
Harbs
> On Nov 3, 2017, at 1:31 PM, Piotr Zarzycki <pi...@gmail.com> wrote:
>
> Hi Harbs,
>
> Fully agree what you are saying! Sometimes I have found myself difficult
> without deeper knowledge how something is working when I check the code.
> One thing which I can mention is that we already have Selenium integrated.
> If you are going to spend some time on that I would start from the point
> [1] and see whether we can use it.
>
> As for your coming to US - Wish I could be there!
>
> [1]
> https://github.com/apache/royale-asjs/tree/develop/examples/examples-integrationtests <https://github.com/apache/royale-asjs/tree/develop/examples/examples-integrationtests>
>
> Piotr
>
>
> 2017-11-03 12:19 GMT+01:00 Harbs <harbs.lists@gmail.com <ma...@gmail.com>>:
>
>> One topic which keeps coming up is better test coverage for Royale.
>>
>> I think this is becoming a critical issue for a few reasons:
>> 1. As we get close to version 1.0 it’s necessary to have good test
>> coverage for confidence of quality and confidence that we don’t introduce
>> recessive bugs.
>> 2. It’s really hard to accept Github pull requests without examining the
>> code VERY well that it does not introduce recessive bugs. CI which runs
>> automated tests could give a preliminary test on pull requests to ensure
>> that they don’t break anything. If the pull requests do break something, it
>> allows the requester to fix the problem with confidence without taking
>> others’ time.
>>
>> I think we need to break up testing into pieces and figure out a strategy
>> to implement automated tests in a way that they are maintainable.
>>
>> Some points:
>> 1. I think integration into something like Travis would be very helpful.
>> 2. I think there’s a Jenkins plugin for building pull requests. Not sure
>> how useful it is.[1]
>> 3. Josh has created a Node.js-compatible test-runner architecture which
>> could be useful for unit tests on parts of the framework which don’t rely
>> on browser features. (i.e. models and the like) He mentioned the
>> possibility of donating it, and I think that would be a very useful feature.
>> 4. For UI and integration tests, there seem to be some pretty cool
>> integrations using Selenium.[2][3]
>> 5. I think the main testing effort needs to be using JS and something like
>> Josh’s testing framework for non-UI pieces and some easy-to-use Selenium
>> integration for visual pieces and integration tests.
>> 6. We probably also want some API endpoints we can test off of for
>> integration testing.
>>
>> I’m willing to invest time into this.
>>
>> It’s going to be a lot of work building out the tests and I think we need
>> a plan for that. My thoughts:
>>
>> 1. Step one is to make it easy to write meaningful automated tests and
>> establish a clear pattern.
>> 2. Step two is to start writing tests starting from the most-used/easiest
>> to beak pieces and work out from there.
>> 3. Once the pattern is established, any new pieces MUST have testing
>> coverage.
>> 4. When fixing bugs, attention should be paid to adding testing for that
>> component.
>> 5. When a pull request comes in on a piece which does not have unit test,
>> a test must be written before accepting the pull request. The test does not
>> need to be written by the requester. Before examining the request, the test
>> should be written to pass for expected behavior and fail for the bug that
>> the pull request is attempting to fix (assuming the pull request is to fix
>> a bug).
>>
>> Thoughts?
>> Harbs
>>
>> P.S. I’m thinking of coming to the US in late December/early January. I
>> would be interested in getting together for a hacking session with folks
>> who are available.
>>
>> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
>> request+builder+plugin <https://wiki.jenkins.io/ <https://wiki.jenkins.io/>
>> display/JENKINS/GitHub+pull+request+builder+plugin>
>> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/> <
>> https://docs.travis-ci.com/user/gui-and-headless-browsers/ <https://docs.travis-ci.com/user/gui-and-headless-browsers/>>
>> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/products/integrations> <https://saucelabs.com/ <https://saucelabs.com/>
>> products/integrations>
>
>
>
>
> --
>
> Piotr Zarzycki
>
> mobile: +48 880 859 557
> skype: zarzycki10
>
> LinkedIn: http://www.linkedin.com/piotrzarzycki <http://www.linkedin.com/piotrzarzycki>
> <https://pl.linkedin.com/in/piotr-zarzycki-92a53552 <https://pl.linkedin.com/in/piotr-zarzycki-92a53552>>
>
> GitHub: https://github.com/piotrzarzycki21 <https://github.com/piotrzarzycki21>
Re: Unit Tests et. al.
Posted by Piotr Zarzycki <pi...@gmail.com>.
Hi Harbs,
Fully agree what you are saying! Sometimes I have found myself difficult
without deeper knowledge how something is working when I check the code.
One thing which I can mention is that we already have Selenium integrated.
If you are going to spend some time on that I would start from the point
[1] and see whether we can use it.
As for your coming to US - Wish I could be there!
[1]
https://github.com/apache/royale-asjs/tree/develop/examples/examples-integrationtests
Piotr
2017-11-03 12:19 GMT+01:00 Harbs <ha...@gmail.com>:
> One topic which keeps coming up is better test coverage for Royale.
>
> I think this is becoming a critical issue for a few reasons:
> 1. As we get close to version 1.0 it’s necessary to have good test
> coverage for confidence of quality and confidence that we don’t introduce
> recessive bugs.
> 2. It’s really hard to accept Github pull requests without examining the
> code VERY well that it does not introduce recessive bugs. CI which runs
> automated tests could give a preliminary test on pull requests to ensure
> that they don’t break anything. If the pull requests do break something, it
> allows the requester to fix the problem with confidence without taking
> others’ time.
>
> I think we need to break up testing into pieces and figure out a strategy
> to implement automated tests in a way that they are maintainable.
>
> Some points:
> 1. I think integration into something like Travis would be very helpful.
> 2. I think there’s a Jenkins plugin for building pull requests. Not sure
> how useful it is.[1]
> 3. Josh has created a Node.js-compatible test-runner architecture which
> could be useful for unit tests on parts of the framework which don’t rely
> on browser features. (i.e. models and the like) He mentioned the
> possibility of donating it, and I think that would be a very useful feature.
> 4. For UI and integration tests, there seem to be some pretty cool
> integrations using Selenium.[2][3]
> 5. I think the main testing effort needs to be using JS and something like
> Josh’s testing framework for non-UI pieces and some easy-to-use Selenium
> integration for visual pieces and integration tests.
> 6. We probably also want some API endpoints we can test off of for
> integration testing.
>
> I’m willing to invest time into this.
>
> It’s going to be a lot of work building out the tests and I think we need
> a plan for that. My thoughts:
>
> 1. Step one is to make it easy to write meaningful automated tests and
> establish a clear pattern.
> 2. Step two is to start writing tests starting from the most-used/easiest
> to beak pieces and work out from there.
> 3. Once the pattern is established, any new pieces MUST have testing
> coverage.
> 4. When fixing bugs, attention should be paid to adding testing for that
> component.
> 5. When a pull request comes in on a piece which does not have unit test,
> a test must be written before accepting the pull request. The test does not
> need to be written by the requester. Before examining the request, the test
> should be written to pass for expected behavior and fail for the bug that
> the pull request is attempting to fix (assuming the pull request is to fix
> a bug).
>
> Thoughts?
> Harbs
>
> P.S. I’m thinking of coming to the US in late December/early January. I
> would be interested in getting together for a hacking session with folks
> who are available.
>
> [1]https://wiki.jenkins.io/display/JENKINS/GitHub+pull+
> request+builder+plugin <https://wiki.jenkins.io/
> display/JENKINS/GitHub+pull+request+builder+plugin>
> [2]https://docs.travis-ci.com/user/gui-and-headless-browsers/ <
> https://docs.travis-ci.com/user/gui-and-headless-browsers/>
> [3]https://saucelabs.com/products/integrations <https://saucelabs.com/
> products/integrations>
--
Piotr Zarzycki
mobile: +48 880 859 557
skype: zarzycki10
LinkedIn: http://www.linkedin.com/piotrzarzycki
<https://pl.linkedin.com/in/piotr-zarzycki-92a53552>
GitHub: https://github.com/piotrzarzycki21