You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flex.apache.org by Peter Ent <pe...@adobe.com> on 2012/08/24 20:54:26 UTC
Mustella Test Repairs
I've completed components/DataGrid. I wound up repairing a number of tests due to changes in (x,y) cursor position and in a couple of cases, the wrong image asset was being used. If anyone on a Mac could re-test components/DataGrid after syncing to the development branch, that would be helpful to see how things compare.
./mini_run.sh tests/components/DataGrid/
Thanks,
Peter
Re: Mustella Test Repairs
Posted by Alex Harui <ah...@adobe.com>.
On 8/27/12 7:35 AM, "Carlos Rovira" <ca...@codeoscopic.com> wrote:
> Hi Peter,
>
> yes, I'm fully synchronized with develop branch.
>
> The differences in number of test reported is already scaring me
> since. I saw this before in other folders and is nothing good...
>
> I think you are ok with getting some method to work over since I think
> the team working right now on this is not already using the same
> rules. For example I saw Alex said to Jose to not run -createImages,
> but I think you and I consesuated to use it...we should something
> unified and all people working on it should get over that rules.
>
I'm not sure we can write an algorithm for everyone to follow, but my
general flow is this:
1) Run a folder of tests.
2) Fix the non-CompareBitmap failures by modifying the tests.
3) If I see CompareBitmap failures, use ImageDiffAIR to examine the
differences
a) If the only differences appear the be that the baseline did not use
embedded fonts, then cut a new baseline
b) Otherwise, try to figure out why there are differences and determine
if the test needs fixing or if it is ok to cut a new baseline.
The last part is a loosely defined. But I would assume that, other than
embedded font errors, the baselines checked in are accurate and differences
are important.
If there are just a few differences in the antialiasing, you can try to get
around that by bumping up the colorVariance options on CompareBitmap, but
don't just stick in high numbers to make the test pass.
It is great to see all of your efforts and I know we are in a hurry to get
tests passing, but we don't want to skip over important tests to reach that
goal.
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
Re: Mustella Test Repairs
Posted by Carlos Rovira <ca...@codeoscopic.com>.
Hi Peter,
yes, I'm fully synchronized with develop branch.
The differences in number of test reported is already scaring me
since. I saw this before in other folders and is nothing good...
I think you are ok with getting some method to work over since I think
the team working right now on this is not already using the same
rules. For example I saw Alex said to Jose to not run -createImages,
but I think you and I consesuated to use it...we should something
unified and all people working on it should get over that rules.
2012/8/27 Peter Ent <pe...@adobe.com>:
> Thanks. So we have two issues.
>
> When I run components/DataGrid, I now consistently get 1293 passes and 0
> failures. You are seeing only 873 passes. I wonder what happened to the
> other 420 tests? Did you sync with [1]?
>
> We are concerned that images creates on system A will not match on system
> B and it looks like that may be the case here. We are trying to get 100%
> of the tests to work and pass for everyone.
>
> Maybe the thing to do is to take a single, but comprehensive, set of
> tests, and figure out how to get 100% of them to work for everyone and in
> doing so, apply what we learn to the rest of the mustella tests.
>
> --peter
>
> [1]
> https://svn.apache.org/repos/asf/incubator/flex/sdk/branches/develop/mustel
> la/tests/components/DataGrid
>
> On 8/27/12 10:10 AM, "Carlos Rovira" <ca...@codeoscopic.com> wrote:
>
>>Hi Peter,
>>
>> ./mini_run.sh tests/components/DataGrid/ does not work for me. It
>>reports 21 failures. Then reruning with -createImages flag it passed:
>>
>>[java] =====================================================
>> [java] Passes: 873
>> [java] Fails: 0
>> [java] =====================================================
>>
>>I'm on Mac 10.8
>>
>>
>>
>>2012/8/24 Peter Ent <pe...@adobe.com>:
>>> I've completed components/DataGrid. I wound up repairing a number of
>>>tests due to changes in (x,y) cursor position and in a couple of cases,
>>>the wrong image asset was being used. If anyone on a Mac could re-test
>>>components/DataGrid after syncing to the development branch, that would
>>>be helpful to see how things compare.
>>>
>>> ./mini_run.sh tests/components/DataGrid/
>>>
>>> Thanks,
>>> Peter
>>
>>
>>
>>--
>>Carlos Rovira
>>Director de Tecnología
>>M: +34 607 22 60 05
>>F: +34 912 35 57 77
>>CODEOSCOPIC S.A.
>>Avd. del General Perón, 32
>>Planta 10, Puertas P-Q
>>28020 Madrid
>
--
Carlos Rovira
Director de Tecnología
M: +34 607 22 60 05
F: +34 912 35 57 77
CODEOSCOPIC S.A.
Avd. del General Perón, 32
Planta 10, Puertas P-Q
28020 Madrid
Re: Mustella Test Repairs
Posted by Alex Harui <ah...@adobe.com>.
On 8/27/12 8:25 AM, "Carlos Rovira" <ca...@codeoscopic.com> wrote:
Hi Carlos, thanks for plugging away at this.
> To fill start filling the gaps ;)
>
> ./mini_run.sh tests/browser/
>
> [java] =====================================================
> [java] Passes: 0
> [java] Fails: 0
> [java] =====================================================
>
> so all OK (all should be in exclusion)
The browser tests probably require a change of configuration in
local.properties and possibly some new work to copy in SWFObject. You will
also have to setup an HTTP server to point to the tests directory.
>
> ./mini_run.sh tests/CompiletimeLocalization/
>
> [java] =====================================================
> [java] Passes: 0
> [java] Fails: 0
> [java] =====================================================
>
> again all OK (all should be in exclusion)
If 0 tests are running, we need to understand why.
--
Alex Harui
Flex SDK Team
Adobe Systems, Inc.
http://blogs.adobe.com/aharui
Re: Mustella Test Repairs
Posted by Carlos Rovira <ca...@codeoscopic.com>.
To fill start filling the gaps ;)
./mini_run.sh tests/browser/
[java] =====================================================
[java] Passes: 0
[java] Fails: 0
[java] =====================================================
so all OK (all should be in exclusion)
./mini_run.sh tests/CompiletimeLocalization/
[java] =====================================================
[java] Passes: 0
[java] Fails: 0
[java] =====================================================
again all OK (all should be in exclusion)
Best,
Carlos
2012/8/27 Carlos Rovira <ca...@codeoscopic.com>:
> Great Peter! this now is more easy to work with! :)
>
> 2012/8/27 Peter Ent <pe...@adobe.com>:
>> I've updated the Mustella wiki page and sorted the table (I put in some
>> "sub headers" for those test sections that are currently being broken out
>> due to their large size - maybe they should get their own table - I might
>> do that later). I culled all the emails and I think I got everything;
>> please let me know if I missed something.
>>
>> I think the table needs a better status column than notes: Passed (you ran
>> mini_run and all tests passed, so there is nothing to fix), In Progress
>> (you are fixing the test group), Patch Ready (+JIRA bug number), Patch
>> Evaluating (by a committer), Patch Submitted (things checked out, you can
>> sync and tests should work for everyone), and Returned (meaning, the
>> committer could not get the patch to work and/or tests still fail for the
>> committer, so it is being returned for further work).
>>
>> --peter
>>
>> On 8/27/12 11:04 AM, "Carlos Rovira" <ca...@codeoscopic.com> wrote:
>>
>>>Yes, I'm having a great senssation with people supporting the project.
>>>Is great to see it :)
>>>
>>>
>>>2012/8/27 Peter Ent <pe...@adobe.com>:
>>>> I'm in the process of gathering all of the info from this weekend. There
>>>> was a lot of activity, which is great to see.
>>>> --peter
>>>>
>>>> On 8/27/12 10:36 AM, "Carlos Rovira" <ca...@codeoscopic.com>
>>>>wrote:
>>>>
>>>>>Hi Carol,
>>>>>
>>>>>just Jose told me about that. Yes, it would be helpful.
>>>>>
>>>>>we need to update it with the latest info from this weekend and point
>>>>>where we are each one.
>>>>>
>>>>>Thanks
>>>>>
>>>>>2012/8/27 Carol Frampton <cf...@adobe.com>:
>>>>>> Would it be ok if I alphabetize the Mustella Test Journal section of
>>>>>> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
>>>>>>rather
>>>>>> than leave it in chronological order? It is getting hard to tell what
>>>>>>has
>>>>>> been done.
>>>>>>
>>>>>> Carol
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>--
>>>>>Carlos Rovira
>>>>>Director de Tecnología
>>>>>M: +34 607 22 60 05
>>>>>F: +34 912 35 57 77
>>>>>CODEOSCOPIC S.A.
>>>>>Avd. del General Perón, 32
>>>>>Planta 10, Puertas P-Q
>>>>>28020 Madrid
>>>>
>>>
>>>
>>>
>>>--
>>>Carlos Rovira
>>>Director de Tecnología
>>>M: +34 607 22 60 05
>>>F: +34 912 35 57 77
>>>CODEOSCOPIC S.A.
>>>Avd. del General Perón, 32
>>>Planta 10, Puertas P-Q
>>>28020 Madrid
>>
>
>
>
> --
> Carlos Rovira
> Director de Tecnología
> M: +34 607 22 60 05
> F: +34 912 35 57 77
> CODEOSCOPIC S.A.
> Avd. del General Perón, 32
> Planta 10, Puertas P-Q
> 28020 Madrid
--
Carlos Rovira
Director de Tecnología
M: +34 607 22 60 05
F: +34 912 35 57 77
CODEOSCOPIC S.A.
Avd. del General Perón, 32
Planta 10, Puertas P-Q
28020 Madrid
Re: Mustella Test Repairs
Posted by Carlos Rovira <ca...@codeoscopic.com>.
Great Peter! this now is more easy to work with! :)
2012/8/27 Peter Ent <pe...@adobe.com>:
> I've updated the Mustella wiki page and sorted the table (I put in some
> "sub headers" for those test sections that are currently being broken out
> due to their large size - maybe they should get their own table - I might
> do that later). I culled all the emails and I think I got everything;
> please let me know if I missed something.
>
> I think the table needs a better status column than notes: Passed (you ran
> mini_run and all tests passed, so there is nothing to fix), In Progress
> (you are fixing the test group), Patch Ready (+JIRA bug number), Patch
> Evaluating (by a committer), Patch Submitted (things checked out, you can
> sync and tests should work for everyone), and Returned (meaning, the
> committer could not get the patch to work and/or tests still fail for the
> committer, so it is being returned for further work).
>
> --peter
>
> On 8/27/12 11:04 AM, "Carlos Rovira" <ca...@codeoscopic.com> wrote:
>
>>Yes, I'm having a great senssation with people supporting the project.
>>Is great to see it :)
>>
>>
>>2012/8/27 Peter Ent <pe...@adobe.com>:
>>> I'm in the process of gathering all of the info from this weekend. There
>>> was a lot of activity, which is great to see.
>>> --peter
>>>
>>> On 8/27/12 10:36 AM, "Carlos Rovira" <ca...@codeoscopic.com>
>>>wrote:
>>>
>>>>Hi Carol,
>>>>
>>>>just Jose told me about that. Yes, it would be helpful.
>>>>
>>>>we need to update it with the latest info from this weekend and point
>>>>where we are each one.
>>>>
>>>>Thanks
>>>>
>>>>2012/8/27 Carol Frampton <cf...@adobe.com>:
>>>>> Would it be ok if I alphabetize the Mustella Test Journal section of
>>>>> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
>>>>>rather
>>>>> than leave it in chronological order? It is getting hard to tell what
>>>>>has
>>>>> been done.
>>>>>
>>>>> Carol
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>--
>>>>Carlos Rovira
>>>>Director de Tecnología
>>>>M: +34 607 22 60 05
>>>>F: +34 912 35 57 77
>>>>CODEOSCOPIC S.A.
>>>>Avd. del General Perón, 32
>>>>Planta 10, Puertas P-Q
>>>>28020 Madrid
>>>
>>
>>
>>
>>--
>>Carlos Rovira
>>Director de Tecnología
>>M: +34 607 22 60 05
>>F: +34 912 35 57 77
>>CODEOSCOPIC S.A.
>>Avd. del General Perón, 32
>>Planta 10, Puertas P-Q
>>28020 Madrid
>
--
Carlos Rovira
Director de Tecnología
M: +34 607 22 60 05
F: +34 912 35 57 77
CODEOSCOPIC S.A.
Avd. del General Perón, 32
Planta 10, Puertas P-Q
28020 Madrid
Re: Mustella Test Repairs
Posted by Peter Ent <pe...@adobe.com>.
I've updated the Mustella wiki page and sorted the table (I put in some
"sub headers" for those test sections that are currently being broken out
due to their large size - maybe they should get their own table - I might
do that later). I culled all the emails and I think I got everything;
please let me know if I missed something.
I think the table needs a better status column than notes: Passed (you ran
mini_run and all tests passed, so there is nothing to fix), In Progress
(you are fixing the test group), Patch Ready (+JIRA bug number), Patch
Evaluating (by a committer), Patch Submitted (things checked out, you can
sync and tests should work for everyone), and Returned (meaning, the
committer could not get the patch to work and/or tests still fail for the
committer, so it is being returned for further work).
--peter
On 8/27/12 11:04 AM, "Carlos Rovira" <ca...@codeoscopic.com> wrote:
>Yes, I'm having a great senssation with people supporting the project.
>Is great to see it :)
>
>
>2012/8/27 Peter Ent <pe...@adobe.com>:
>> I'm in the process of gathering all of the info from this weekend. There
>> was a lot of activity, which is great to see.
>> --peter
>>
>> On 8/27/12 10:36 AM, "Carlos Rovira" <ca...@codeoscopic.com>
>>wrote:
>>
>>>Hi Carol,
>>>
>>>just Jose told me about that. Yes, it would be helpful.
>>>
>>>we need to update it with the latest info from this weekend and point
>>>where we are each one.
>>>
>>>Thanks
>>>
>>>2012/8/27 Carol Frampton <cf...@adobe.com>:
>>>> Would it be ok if I alphabetize the Mustella Test Journal section of
>>>> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
>>>>rather
>>>> than leave it in chronological order? It is getting hard to tell what
>>>>has
>>>> been done.
>>>>
>>>> Carol
>>>>
>>>>
>>>
>>>
>>>
>>>--
>>>Carlos Rovira
>>>Director de Tecnología
>>>M: +34 607 22 60 05
>>>F: +34 912 35 57 77
>>>CODEOSCOPIC S.A.
>>>Avd. del General Perón, 32
>>>Planta 10, Puertas P-Q
>>>28020 Madrid
>>
>
>
>
>--
>Carlos Rovira
>Director de Tecnología
>M: +34 607 22 60 05
>F: +34 912 35 57 77
>CODEOSCOPIC S.A.
>Avd. del General Perón, 32
>Planta 10, Puertas P-Q
>28020 Madrid
Re: Mustella Test Repairs
Posted by Carlos Rovira <ca...@codeoscopic.com>.
Yes, I'm having a great senssation with people supporting the project.
Is great to see it :)
2012/8/27 Peter Ent <pe...@adobe.com>:
> I'm in the process of gathering all of the info from this weekend. There
> was a lot of activity, which is great to see.
> --peter
>
> On 8/27/12 10:36 AM, "Carlos Rovira" <ca...@codeoscopic.com> wrote:
>
>>Hi Carol,
>>
>>just Jose told me about that. Yes, it would be helpful.
>>
>>we need to update it with the latest info from this weekend and point
>>where we are each one.
>>
>>Thanks
>>
>>2012/8/27 Carol Frampton <cf...@adobe.com>:
>>> Would it be ok if I alphabetize the Mustella Test Journal section of
>>> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
>>>rather
>>> than leave it in chronological order? It is getting hard to tell what
>>>has
>>> been done.
>>>
>>> Carol
>>>
>>>
>>
>>
>>
>>--
>>Carlos Rovira
>>Director de Tecnología
>>M: +34 607 22 60 05
>>F: +34 912 35 57 77
>>CODEOSCOPIC S.A.
>>Avd. del General Perón, 32
>>Planta 10, Puertas P-Q
>>28020 Madrid
>
--
Carlos Rovira
Director de Tecnología
M: +34 607 22 60 05
F: +34 912 35 57 77
CODEOSCOPIC S.A.
Avd. del General Perón, 32
Planta 10, Puertas P-Q
28020 Madrid
Re: Mustella Test Repairs
Posted by Peter Ent <pe...@adobe.com>.
I'm in the process of gathering all of the info from this weekend. There
was a lot of activity, which is great to see.
--peter
On 8/27/12 10:36 AM, "Carlos Rovira" <ca...@codeoscopic.com> wrote:
>Hi Carol,
>
>just Jose told me about that. Yes, it would be helpful.
>
>we need to update it with the latest info from this weekend and point
>where we are each one.
>
>Thanks
>
>2012/8/27 Carol Frampton <cf...@adobe.com>:
>> Would it be ok if I alphabetize the Mustella Test Journal section of
>> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview
>>rather
>> than leave it in chronological order? It is getting hard to tell what
>>has
>> been done.
>>
>> Carol
>>
>>
>
>
>
>--
>Carlos Rovira
>Director de Tecnología
>M: +34 607 22 60 05
>F: +34 912 35 57 77
>CODEOSCOPIC S.A.
>Avd. del General Perón, 32
>Planta 10, Puertas P-Q
>28020 Madrid
Re: Mustella Test Repairs
Posted by Carlos Rovira <ca...@codeoscopic.com>.
Hi Carol,
just Jose told me about that. Yes, it would be helpful.
we need to update it with the latest info from this weekend and point
where we are each one.
Thanks
2012/8/27 Carol Frampton <cf...@adobe.com>:
> Would it be ok if I alphabetize the Mustella Test Journal section of
> https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview rather
> than leave it in chronological order? It is getting hard to tell what has
> been done.
>
> Carol
>
>
--
Carlos Rovira
Director de Tecnología
M: +34 607 22 60 05
F: +34 912 35 57 77
CODEOSCOPIC S.A.
Avd. del General Perón, 32
Planta 10, Puertas P-Q
28020 Madrid
Re: Mustella Test Repairs
Posted by Peter Ent <pe...@adobe.com>.
That's a good idea. I have it open for edit right now so I'll do that.
--peter
On 8/27/12 10:32 AM, "Carol Frampton" <cf...@adobe.com> wrote:
>Would it be ok if I alphabetize the Mustella Test Journal section of
>https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview rather
>than leave it in chronological order? It is getting hard to tell what has
>been done.
>
>Carol
>
>
Re: Mustella Test Repairs
Posted by Carol Frampton <cf...@adobe.com>.
Would it be ok if I alphabetize the Mustella Test Journal section of
https://cwiki.apache.org/confluence/display/FLEX/Mustella+Overview rather
than leave it in chronological order? It is getting hard to tell what has
been done.
Carol
Re: Mustella Test Repairs
Posted by Peter Ent <pe...@adobe.com>.
Thanks. So we have two issues.
When I run components/DataGrid, I now consistently get 1293 passes and 0
failures. You are seeing only 873 passes. I wonder what happened to the
other 420 tests? Did you sync with [1]?
We are concerned that images creates on system A will not match on system
B and it looks like that may be the case here. We are trying to get 100%
of the tests to work and pass for everyone.
Maybe the thing to do is to take a single, but comprehensive, set of
tests, and figure out how to get 100% of them to work for everyone and in
doing so, apply what we learn to the rest of the mustella tests.
--peter
[1]
https://svn.apache.org/repos/asf/incubator/flex/sdk/branches/develop/mustel
la/tests/components/DataGrid
On 8/27/12 10:10 AM, "Carlos Rovira" <ca...@codeoscopic.com> wrote:
>Hi Peter,
>
> ./mini_run.sh tests/components/DataGrid/ does not work for me. It
>reports 21 failures. Then reruning with -createImages flag it passed:
>
>[java] =====================================================
> [java] Passes: 873
> [java] Fails: 0
> [java] =====================================================
>
>I'm on Mac 10.8
>
>
>
>2012/8/24 Peter Ent <pe...@adobe.com>:
>> I've completed components/DataGrid. I wound up repairing a number of
>>tests due to changes in (x,y) cursor position and in a couple of cases,
>>the wrong image asset was being used. If anyone on a Mac could re-test
>>components/DataGrid after syncing to the development branch, that would
>>be helpful to see how things compare.
>>
>> ./mini_run.sh tests/components/DataGrid/
>>
>> Thanks,
>> Peter
>
>
>
>--
>Carlos Rovira
>Director de Tecnología
>M: +34 607 22 60 05
>F: +34 912 35 57 77
>CODEOSCOPIC S.A.
>Avd. del General Perón, 32
>Planta 10, Puertas P-Q
>28020 Madrid
Re: Mustella Test Repairs
Posted by Carlos Rovira <ca...@codeoscopic.com>.
Hi Peter,
./mini_run.sh tests/components/DataGrid/ does not work for me. It
reports 21 failures. Then reruning with -createImages flag it passed:
[java] =====================================================
[java] Passes: 873
[java] Fails: 0
[java] =====================================================
I'm on Mac 10.8
2012/8/24 Peter Ent <pe...@adobe.com>:
> I've completed components/DataGrid. I wound up repairing a number of tests due to changes in (x,y) cursor position and in a couple of cases, the wrong image asset was being used. If anyone on a Mac could re-test components/DataGrid after syncing to the development branch, that would be helpful to see how things compare.
>
> ./mini_run.sh tests/components/DataGrid/
>
> Thanks,
> Peter
--
Carlos Rovira
Director de Tecnología
M: +34 607 22 60 05
F: +34 912 35 57 77
CODEOSCOPIC S.A.
Avd. del General Perón, 32
Planta 10, Puertas P-Q
28020 Madrid