You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@echarts.apache.org by Yi Shen <sh...@gmail.com> on 2019/09/03 10:50:58 UTC
[INFO] Add tool to run all tests automatically
Hi all,
I opened a pull request about adding a tool for running all
tests automatically
https://github.com/apache/incubator-echarts/pull/11154
Please leave your comments if you have any suggestions. Thanks
--
Yi Shen
Senior Developer
Baidu, Inc.
Re: [INFO] Add tool to run all tests automatically
Posted by SHUANG SU <su...@gmail.com>.
Just express my surprise and delight after tried this tool.
Beyond that feature that we need for a long time, it provides fine great
details!
------------------------------
Su Shuang (100pah)
------------------------------
On Thu, 12 Sep 2019 at 02:41, Yi Shen <sh...@gmail.com> wrote:
> Hi all, here is some updated progress about the visual regression testing
> tool.
>
> - Added interaction testing. It can replay recorded drag, click, select
> actions and compare the screenshots each action finished.
> - Support multi-threaded testing. Now it can be up to 4x faster when
> having 8 threads running all the tests rather than 1.
> - Support both SVG / Canvas renderer testing.
> - Support running and comparing any two selected versions.
> - Better UI
>
> Here is a screenshot of the dashboard
>
> [image: MacHi 2019-09-12 02-29-19.jpg]
>
> I think it's almost ready for regular testing now. Try with the following
> steps:
>
> - Checkout the latest code of master branch
> - `npm install && npm install puppeteer` to dependencies
> - `npm run test:visual` to start the server and open the dashboard
> - Select tests on the left and run!
>
> Please give your feedback if you have any problems using this. More
> detailed tutorial is on the way:)
>
> On Thu, Sep 5, 2019 at 11:59 AM Clement Ho <cl...@gitlab.com.invalid>
> wrote:
>
>> Thanks for the in-depth explanation!
>>
>> On Wed, Sep 4, 2019 at 10:35 PM Yi Shen <sh...@gmail.com> wrote:
>>
>> > Hi Clement.
>> >
>> > The problem we are trying to solve by this testing tool is:
>> > Running all the exists HTML test cases[1] automatically and compare
>> current
>> > developing version to the previous stable release to test if any
>> function
>> > is broken.
>> > It's more like regression testing.
>> >
>> > Jest and jest-image-snapshot is my first option.
>> > But jest-image-snap is designed to compare with the pre-generated
>> baseline
>> > snapshots. For us, there are some problems:
>> >
>> > 1. Not like normal tests, the pre-generated snapshots are depend on
>> > puppeteer versions, operating systems, or even hardwares.
>> > For example the default system font on macOS and windows are different.
>> > These are all unnecessary variables in a testing system, which we should
>> > eliminate.
>> > 2. Baseline snapshots for nearly 400 tests may have a huge size.
>> > 3. There is also a small problem, we need to update the baseline
>> snapshots
>> > when the test cases are changed.
>> >
>> > There are some workarounds on this issue, like developers generate his
>> own
>> > baseline snapshots, instead of uploading the baseline snapshots.
>> > But I think a better solution for us is picking out the underlying
>> > pixelmatch[2] and rewrite the testing module, which will do:
>> > 1. Download the latest stable ECharts release.
>> > 2. Run test cases with both stable release and developing version to get
>> > snapshots, in a consistent puppeeter, OS, and hardware.
>> > 3. Compare the snapshots and check if there is any diff.
>> >
>> > At last, here are some downsides about this testing tool:
>> > 1. It can't test new add functions. But it's mainly for regression
>> testing
>> > so I think it's not a problem.
>> > 2. Running each test case twice may be time time consuming.
>> >
>> > [1] https://github.com/apache/incubator-echarts/tree/master/test
>> > [2] https://github.com/mapbox/pixelmatch
>> >
>> > On Thu, Sep 5, 2019 at 8:48 AM Clement Ho <cl...@gitlab.com.invalid>
>> > wrote:
>> >
>> > > Curious as to the decision not to use something like jest image
>> > snapshots.
>> > > Could you share your thought process?
>> > >
>> > > On Tue, Sep 3, 2019 at 7:23 AM Yi Shen <sh...@gmail.com> wrote:
>> > >
>> > > > Thanks Ovilia.
>> > > >
>> > > > The progress and todo is in the pull request description. The next
>> big
>> > > > thing is supporting interaction testing in the tool.
>> > > >
>> > > > On Tue, Sep 3, 2019 at 8:17 PM Ovilia <ov...@gmail.com>
>> wrote:
>> > > >
>> > > > > Hi Shen,
>> > > > >
>> > > > > Very nice work!
>> > > > > Can we know more about the next plan about the test framework?
>> > > > >
>> > > > > Wenli
>> > > > >
>> > > > >
>> > > > > On Tue, Sep 3, 2019 at 8:05 PM SHUANG SU <su...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > Wonderful!
>> > > > > >
>> > > > > > ------------------------------
>> > > > > > Su Shuang (100pah)
>> > > > > > ------------------------------
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > On Tue, 3 Sep 2019 at 18:51, Yi Shen <sh...@gmail.com>
>> wrote:
>> > > > > >
>> > > > > > > Hi all,
>> > > > > > >
>> > > > > > > I opened a pull request about adding a tool for running all
>> > > > > > > tests automatically
>> > > > > > >
>> > > > > > > https://github.com/apache/incubator-echarts/pull/11154
>> > > > > > >
>> > > > > > > Please leave your comments if you have any suggestions. Thanks
>> > > > > > >
>> > > > > > > --
>> > > > > > > Yi Shen
>> > > > > > > Senior Developer
>> > > > > > > Baidu, Inc.
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > > >
>> > > > --
>> > > > Yi Shen
>> > > > Senior Developer
>> > > > Baidu, Inc.
>> > > >
>> > >
>> >
>> >
>> > --
>> > Yi Shen
>> > Senior Developer
>> > Baidu, Inc.
>> >
>>
>
>
> --
> Yi Shen
> Senior Developer
> Baidu, Inc.
>
Re: [INFO] Add tool to run all tests automatically
Posted by Yi Shen <sh...@gmail.com>.
Hi all, here is some updated progress about the visual regression testing
tool.
- Added interaction testing. It can replay recorded drag, click, select
actions and compare the screenshots each action finished.
- Support multi-threaded testing. Now it can be up to 4x faster when
having 8 threads running all the tests rather than 1.
- Support both SVG / Canvas renderer testing.
- Support running and comparing any two selected versions.
- Better UI
Here is a screenshot of the dashboard
[image: MacHi 2019-09-12 02-29-19.jpg]
I think it's almost ready for regular testing now. Try with the following
steps:
- Checkout the latest code of master branch
- `npm install && npm install puppeteer` to dependencies
- `npm run test:visual` to start the server and open the dashboard
- Select tests on the left and run!
Please give your feedback if you have any problems using this. More
detailed tutorial is on the way:)
On Thu, Sep 5, 2019 at 11:59 AM Clement Ho <cl...@gitlab.com.invalid>
wrote:
> Thanks for the in-depth explanation!
>
> On Wed, Sep 4, 2019 at 10:35 PM Yi Shen <sh...@gmail.com> wrote:
>
> > Hi Clement.
> >
> > The problem we are trying to solve by this testing tool is:
> > Running all the exists HTML test cases[1] automatically and compare
> current
> > developing version to the previous stable release to test if any function
> > is broken.
> > It's more like regression testing.
> >
> > Jest and jest-image-snapshot is my first option.
> > But jest-image-snap is designed to compare with the pre-generated
> baseline
> > snapshots. For us, there are some problems:
> >
> > 1. Not like normal tests, the pre-generated snapshots are depend on
> > puppeteer versions, operating systems, or even hardwares.
> > For example the default system font on macOS and windows are different.
> > These are all unnecessary variables in a testing system, which we should
> > eliminate.
> > 2. Baseline snapshots for nearly 400 tests may have a huge size.
> > 3. There is also a small problem, we need to update the baseline
> snapshots
> > when the test cases are changed.
> >
> > There are some workarounds on this issue, like developers generate his
> own
> > baseline snapshots, instead of uploading the baseline snapshots.
> > But I think a better solution for us is picking out the underlying
> > pixelmatch[2] and rewrite the testing module, which will do:
> > 1. Download the latest stable ECharts release.
> > 2. Run test cases with both stable release and developing version to get
> > snapshots, in a consistent puppeeter, OS, and hardware.
> > 3. Compare the snapshots and check if there is any diff.
> >
> > At last, here are some downsides about this testing tool:
> > 1. It can't test new add functions. But it's mainly for regression
> testing
> > so I think it's not a problem.
> > 2. Running each test case twice may be time time consuming.
> >
> > [1] https://github.com/apache/incubator-echarts/tree/master/test
> > [2] https://github.com/mapbox/pixelmatch
> >
> > On Thu, Sep 5, 2019 at 8:48 AM Clement Ho <cl...@gitlab.com.invalid>
> > wrote:
> >
> > > Curious as to the decision not to use something like jest image
> > snapshots.
> > > Could you share your thought process?
> > >
> > > On Tue, Sep 3, 2019 at 7:23 AM Yi Shen <sh...@gmail.com> wrote:
> > >
> > > > Thanks Ovilia.
> > > >
> > > > The progress and todo is in the pull request description. The next
> big
> > > > thing is supporting interaction testing in the tool.
> > > >
> > > > On Tue, Sep 3, 2019 at 8:17 PM Ovilia <ov...@gmail.com> wrote:
> > > >
> > > > > Hi Shen,
> > > > >
> > > > > Very nice work!
> > > > > Can we know more about the next plan about the test framework?
> > > > >
> > > > > Wenli
> > > > >
> > > > >
> > > > > On Tue, Sep 3, 2019 at 8:05 PM SHUANG SU <su...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Wonderful!
> > > > > >
> > > > > > ------------------------------
> > > > > > Su Shuang (100pah)
> > > > > > ------------------------------
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, 3 Sep 2019 at 18:51, Yi Shen <sh...@gmail.com>
> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I opened a pull request about adding a tool for running all
> > > > > > > tests automatically
> > > > > > >
> > > > > > > https://github.com/apache/incubator-echarts/pull/11154
> > > > > > >
> > > > > > > Please leave your comments if you have any suggestions. Thanks
> > > > > > >
> > > > > > > --
> > > > > > > Yi Shen
> > > > > > > Senior Developer
> > > > > > > Baidu, Inc.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > Yi Shen
> > > > Senior Developer
> > > > Baidu, Inc.
> > > >
> > >
> >
> >
> > --
> > Yi Shen
> > Senior Developer
> > Baidu, Inc.
> >
>
--
Yi Shen
Senior Developer
Baidu, Inc.
Re: [INFO] Add tool to run all tests automatically
Posted by Clement Ho <cl...@gitlab.com.INVALID>.
Thanks for the in-depth explanation!
On Wed, Sep 4, 2019 at 10:35 PM Yi Shen <sh...@gmail.com> wrote:
> Hi Clement.
>
> The problem we are trying to solve by this testing tool is:
> Running all the exists HTML test cases[1] automatically and compare current
> developing version to the previous stable release to test if any function
> is broken.
> It's more like regression testing.
>
> Jest and jest-image-snapshot is my first option.
> But jest-image-snap is designed to compare with the pre-generated baseline
> snapshots. For us, there are some problems:
>
> 1. Not like normal tests, the pre-generated snapshots are depend on
> puppeteer versions, operating systems, or even hardwares.
> For example the default system font on macOS and windows are different.
> These are all unnecessary variables in a testing system, which we should
> eliminate.
> 2. Baseline snapshots for nearly 400 tests may have a huge size.
> 3. There is also a small problem, we need to update the baseline snapshots
> when the test cases are changed.
>
> There are some workarounds on this issue, like developers generate his own
> baseline snapshots, instead of uploading the baseline snapshots.
> But I think a better solution for us is picking out the underlying
> pixelmatch[2] and rewrite the testing module, which will do:
> 1. Download the latest stable ECharts release.
> 2. Run test cases with both stable release and developing version to get
> snapshots, in a consistent puppeeter, OS, and hardware.
> 3. Compare the snapshots and check if there is any diff.
>
> At last, here are some downsides about this testing tool:
> 1. It can't test new add functions. But it's mainly for regression testing
> so I think it's not a problem.
> 2. Running each test case twice may be time time consuming.
>
> [1] https://github.com/apache/incubator-echarts/tree/master/test
> [2] https://github.com/mapbox/pixelmatch
>
> On Thu, Sep 5, 2019 at 8:48 AM Clement Ho <cl...@gitlab.com.invalid>
> wrote:
>
> > Curious as to the decision not to use something like jest image
> snapshots.
> > Could you share your thought process?
> >
> > On Tue, Sep 3, 2019 at 7:23 AM Yi Shen <sh...@gmail.com> wrote:
> >
> > > Thanks Ovilia.
> > >
> > > The progress and todo is in the pull request description. The next big
> > > thing is supporting interaction testing in the tool.
> > >
> > > On Tue, Sep 3, 2019 at 8:17 PM Ovilia <ov...@gmail.com> wrote:
> > >
> > > > Hi Shen,
> > > >
> > > > Very nice work!
> > > > Can we know more about the next plan about the test framework?
> > > >
> > > > Wenli
> > > >
> > > >
> > > > On Tue, Sep 3, 2019 at 8:05 PM SHUANG SU <su...@gmail.com>
> > wrote:
> > > >
> > > > > Wonderful!
> > > > >
> > > > > ------------------------------
> > > > > Su Shuang (100pah)
> > > > > ------------------------------
> > > > >
> > > > >
> > > > >
> > > > > On Tue, 3 Sep 2019 at 18:51, Yi Shen <sh...@gmail.com> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I opened a pull request about adding a tool for running all
> > > > > > tests automatically
> > > > > >
> > > > > > https://github.com/apache/incubator-echarts/pull/11154
> > > > > >
> > > > > > Please leave your comments if you have any suggestions. Thanks
> > > > > >
> > > > > > --
> > > > > > Yi Shen
> > > > > > Senior Developer
> > > > > > Baidu, Inc.
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > Yi Shen
> > > Senior Developer
> > > Baidu, Inc.
> > >
> >
>
>
> --
> Yi Shen
> Senior Developer
> Baidu, Inc.
>
Re: [INFO] Add tool to run all tests automatically
Posted by Yi Shen <sh...@gmail.com>.
Hi Clement.
The problem we are trying to solve by this testing tool is:
Running all the exists HTML test cases[1] automatically and compare current
developing version to the previous stable release to test if any function
is broken.
It's more like regression testing.
Jest and jest-image-snapshot is my first option.
But jest-image-snap is designed to compare with the pre-generated baseline
snapshots. For us, there are some problems:
1. Not like normal tests, the pre-generated snapshots are depend on
puppeteer versions, operating systems, or even hardwares.
For example the default system font on macOS and windows are different.
These are all unnecessary variables in a testing system, which we should
eliminate.
2. Baseline snapshots for nearly 400 tests may have a huge size.
3. There is also a small problem, we need to update the baseline snapshots
when the test cases are changed.
There are some workarounds on this issue, like developers generate his own
baseline snapshots, instead of uploading the baseline snapshots.
But I think a better solution for us is picking out the underlying
pixelmatch[2] and rewrite the testing module, which will do:
1. Download the latest stable ECharts release.
2. Run test cases with both stable release and developing version to get
snapshots, in a consistent puppeeter, OS, and hardware.
3. Compare the snapshots and check if there is any diff.
At last, here are some downsides about this testing tool:
1. It can't test new add functions. But it's mainly for regression testing
so I think it's not a problem.
2. Running each test case twice may be time time consuming.
[1] https://github.com/apache/incubator-echarts/tree/master/test
[2] https://github.com/mapbox/pixelmatch
On Thu, Sep 5, 2019 at 8:48 AM Clement Ho <cl...@gitlab.com.invalid>
wrote:
> Curious as to the decision not to use something like jest image snapshots.
> Could you share your thought process?
>
> On Tue, Sep 3, 2019 at 7:23 AM Yi Shen <sh...@gmail.com> wrote:
>
> > Thanks Ovilia.
> >
> > The progress and todo is in the pull request description. The next big
> > thing is supporting interaction testing in the tool.
> >
> > On Tue, Sep 3, 2019 at 8:17 PM Ovilia <ov...@gmail.com> wrote:
> >
> > > Hi Shen,
> > >
> > > Very nice work!
> > > Can we know more about the next plan about the test framework?
> > >
> > > Wenli
> > >
> > >
> > > On Tue, Sep 3, 2019 at 8:05 PM SHUANG SU <su...@gmail.com>
> wrote:
> > >
> > > > Wonderful!
> > > >
> > > > ------------------------------
> > > > Su Shuang (100pah)
> > > > ------------------------------
> > > >
> > > >
> > > >
> > > > On Tue, 3 Sep 2019 at 18:51, Yi Shen <sh...@gmail.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I opened a pull request about adding a tool for running all
> > > > > tests automatically
> > > > >
> > > > > https://github.com/apache/incubator-echarts/pull/11154
> > > > >
> > > > > Please leave your comments if you have any suggestions. Thanks
> > > > >
> > > > > --
> > > > > Yi Shen
> > > > > Senior Developer
> > > > > Baidu, Inc.
> > > > >
> > > >
> > >
> >
> >
> > --
> > Yi Shen
> > Senior Developer
> > Baidu, Inc.
> >
>
--
Yi Shen
Senior Developer
Baidu, Inc.
Re: [INFO] Add tool to run all tests automatically
Posted by Clement Ho <cl...@gitlab.com.INVALID>.
Curious as to the decision not to use something like jest image snapshots.
Could you share your thought process?
On Tue, Sep 3, 2019 at 7:23 AM Yi Shen <sh...@gmail.com> wrote:
> Thanks Ovilia.
>
> The progress and todo is in the pull request description. The next big
> thing is supporting interaction testing in the tool.
>
> On Tue, Sep 3, 2019 at 8:17 PM Ovilia <ov...@gmail.com> wrote:
>
> > Hi Shen,
> >
> > Very nice work!
> > Can we know more about the next plan about the test framework?
> >
> > Wenli
> >
> >
> > On Tue, Sep 3, 2019 at 8:05 PM SHUANG SU <su...@gmail.com> wrote:
> >
> > > Wonderful!
> > >
> > > ------------------------------
> > > Su Shuang (100pah)
> > > ------------------------------
> > >
> > >
> > >
> > > On Tue, 3 Sep 2019 at 18:51, Yi Shen <sh...@gmail.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I opened a pull request about adding a tool for running all
> > > > tests automatically
> > > >
> > > > https://github.com/apache/incubator-echarts/pull/11154
> > > >
> > > > Please leave your comments if you have any suggestions. Thanks
> > > >
> > > > --
> > > > Yi Shen
> > > > Senior Developer
> > > > Baidu, Inc.
> > > >
> > >
> >
>
>
> --
> Yi Shen
> Senior Developer
> Baidu, Inc.
>
Re: [INFO] Add tool to run all tests automatically
Posted by Yi Shen <sh...@gmail.com>.
Thanks Ovilia.
The progress and todo is in the pull request description. The next big
thing is supporting interaction testing in the tool.
On Tue, Sep 3, 2019 at 8:17 PM Ovilia <ov...@gmail.com> wrote:
> Hi Shen,
>
> Very nice work!
> Can we know more about the next plan about the test framework?
>
> Wenli
>
>
> On Tue, Sep 3, 2019 at 8:05 PM SHUANG SU <su...@gmail.com> wrote:
>
> > Wonderful!
> >
> > ------------------------------
> > Su Shuang (100pah)
> > ------------------------------
> >
> >
> >
> > On Tue, 3 Sep 2019 at 18:51, Yi Shen <sh...@gmail.com> wrote:
> >
> > > Hi all,
> > >
> > > I opened a pull request about adding a tool for running all
> > > tests automatically
> > >
> > > https://github.com/apache/incubator-echarts/pull/11154
> > >
> > > Please leave your comments if you have any suggestions. Thanks
> > >
> > > --
> > > Yi Shen
> > > Senior Developer
> > > Baidu, Inc.
> > >
> >
>
--
Yi Shen
Senior Developer
Baidu, Inc.
Re: [INFO] Add tool to run all tests automatically
Posted by Ovilia <ov...@gmail.com>.
Hi Shen,
Very nice work!
Can we know more about the next plan about the test framework?
Wenli
On Tue, Sep 3, 2019 at 8:05 PM SHUANG SU <su...@gmail.com> wrote:
> Wonderful!
>
> ------------------------------
> Su Shuang (100pah)
> ------------------------------
>
>
>
> On Tue, 3 Sep 2019 at 18:51, Yi Shen <sh...@gmail.com> wrote:
>
> > Hi all,
> >
> > I opened a pull request about adding a tool for running all
> > tests automatically
> >
> > https://github.com/apache/incubator-echarts/pull/11154
> >
> > Please leave your comments if you have any suggestions. Thanks
> >
> > --
> > Yi Shen
> > Senior Developer
> > Baidu, Inc.
> >
>
Re: [INFO] Add tool to run all tests automatically
Posted by SHUANG SU <su...@gmail.com>.
Wonderful!
------------------------------
Su Shuang (100pah)
------------------------------
On Tue, 3 Sep 2019 at 18:51, Yi Shen <sh...@gmail.com> wrote:
> Hi all,
>
> I opened a pull request about adding a tool for running all
> tests automatically
>
> https://github.com/apache/incubator-echarts/pull/11154
>
> Please leave your comments if you have any suggestions. Thanks
>
> --
> Yi Shen
> Senior Developer
> Baidu, Inc.
>