You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by kh...@gmail.com,
kh...@gmail.com on 2018/11/09 22:11:19 UTC
Nightly/Weekly tests for examples
Hi MXNet community,
Recently, I and a few other contributors focussed on fixing examples in our repository which were not working out of the box as expected.
https://github.com/apache/incubator-mxnet/issues/12800
https://github.com/apache/incubator-mxnet/issues/11895
https://github.com/apache/incubator-mxnet/pull/13196
Some of the examples failed after API changes and remained uncaught until a user reported the issue. While the community is actively working on fixing it, it might re-occur after few days if we don’t have a proper mechanism to catch regressions.
So, I would like to propose to enable nightly/weekly tests for the examples similar to what we have for tutorials to catch any such regressions. The test could check only basic functionalities/working of the examples. It can run small examples completely whereas it can run long training examples for only few epochs.
Any thoughts from the community? Any other suggestions for fixing the same?
Regards,
Ankit Khedia
Re: Nightly/Weekly tests for examples
Posted by Aaron Markham <aa...@gmail.com>.
Naveen - I agree with you there. Not every example should be moved to
tutorials... just the ones that would be great to have on the site and
to be tested regularly.
I think that having examples of specific APIs will be useful on the website too.
Ankit, there's a cost/benefit analysis for keeping both a command line
example and a notebook. That has maintenance overhead.
A couple of options to debate and apply on a case-by-case basis:
1) Only maintain one or the other.
2) If it's really important, maintain both, but update both codebases
with notes and directions for maintenance, so that if one is found
broken and is getting updated, contributors know that they should
update the other.
Since the CNN Viz example/tutorial is for visualization, I'd lean
towards #1 and keep only the notebook. Then delete the example, but
since it requires a python file from that examples folder as a
utility, move this to a tutorials_utils folder in /docs. That folder
should be organized appropriately and excluded in the docs build
config in conf.py. Look for "exclude_patterns".
My 3 cents... Cheers,
Aaron
On Tue, Nov 13, 2018 at 11:26 AM Khedia, Ankit
<kh...@amazon.com.invalid> wrote:
>
>
> We can start small consolidating few examples which are there in both tutorials/examples folder.
> One of such example is GradCAM implementation:
> In tutorials - https://github.com/apache/incubator-mxnet/blob/master/docs/tutorials/vision/cnn_visualization.md
> In examples- https://github.com/apache/incubator-mxnet/blob/master/example/cnn_visualization
> The only difference is we have a python script in place of notebook in examples folder for this particular example.
> Any thoughts/suggestions?
>
> On the other hand, I agree with Sandeep that there should be some basic testing of examples and we can start with a small set of python examples to begin with.
>
> —Ankit
>
>
> On Nov 13, 2018, at 10:31 AM, Naveen Swamy <mn...@gmail.com>> wrote:
>
> Aaron, IMO tutorials have a specific purpose, to introduce concepts and
> APIs to the users and I think converting examples to tutorials would
> overwhelm the users, we should carefully choose which examples we want to
> turn into tutorials.
>
> I agree that today examples are graveyard of untested code, my suggestion
> is to add some testing to the example when you touch the example - at the
> least to check the functionality. These can be run once a week.
>
>
> On Tue, Nov 13, 2018 at 6:52 AM Aaron Markham <aa...@gmail.com>>
> wrote:
>
> I've been actively promoting moving examples to tutorials during reviews.
> That way they fall under the testing umbrella and get added to the website.
>
> Many times there's not really a great distinction as to why something is in
> the examples folder, other than it's like a graveyard of untested sample
> code.
>
> I would suggest a starting strategy of when doing updates on examples, see
> if with just a little more effort, ask yourself, can it be converted to a
> tutorial?
>
> The last thing CI needs is more flaky tutorial tests, so whatever is done
> here should use the more robust approaches that are being discussed.
>
> Cheers,
> Aaron
>
> On Mon, Nov 12, 2018, 16:24 sandeep krishnamurthy <
> sandeep.krishna98@gmail.com<ma...@gmail.com> wrote:
>
> Thanks, Ankit for bringing this up. @Anirudh - All the concerns you
> raised
> are very valid. Here are my thoughts:
> 1. There were several examples that were crashing or had compiler errors.
> This is a very bad user experience. All example scripts should be at
> least runnable!
> 2. While I agree examples are too diverse (python scripts, notebooks,
> epochs, print statements etc..) We can always start small, we can start
> with 5 examples. We can use this to streamline all examples to be python
> scripts, print statements, with the main function invoker that can take
> params like epoch, dataset etc.
> 3. We can start with running weekly tests to avoid too long nightly test
> pipeline.
> 4. One possible issue can be on a few examples that depend on a large or
> controlled dataset. I am not sure yet, how to solve this, but, we can
> think.
>
> Any suggestions?
> Best,
> Sandeep
>
>
>
> On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya <an...@gmail.com>>
> wrote:
>
> Hi Ankit,
>
> I have a few concerns about testing examples. Before writing tests for
> examples,
>
> - you will need to first decide what constitutes a test for an
> example,
> because examples are not API calls, which will have return
> statements
> and
> the test can just call the API and assert for certain values. Just
> testing
> if an example is a compilable python script will not add much value
> in
> my
> opinion.
> - And testing for example output and results will require a re-write
> of
> many of the examples, because many of them currently just have print
> statements as outputs and does not return any value as such. I am
> not
> sure
> if it is worth the dev-effort.
> - the current set of examples in the mxnet repo are very diverse -
> some
> are written as python notebooks, some are just python scripts with
> paper
> implementations, and some are just illustrations of certain mxnet
> features.
> I am curious to know how you will write tests for these things.
>
>
> Looking forward to seeing the design of this test bed/framework.
>
>
> Thanks
> Anirudh Acharya
>
> On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
> <ma...@googlemail.com.invalid>> wrote:
>
> Hello Ankit,
>
> that's a great idea! Using the tutorial tests as reference is a great
> starting point. If you are interested, please don't hesitate to
> attend
> the
> Berlin user group in case you would like to discuss your first
> thoughts
> in-person before drafting a design.
>
> -Marco
>
>
> Am Fr., 9. Nov. 2018, 23:23 hat khedia.ankit@gmail.com<ma...@gmail.com> <
> khedia.ankit@gmail.com<ma...@gmail.com>> geschrieben:
>
> Hi MXNet community,
>
> Recently, I and a few other contributors focussed on fixing
> examples
> in
> our repository which were not working out of the box as expected.
> https://github.com/apache/incubator-mxnet/issues/12800
> https://github.com/apache/incubator-mxnet/issues/11895
> https://github.com/apache/incubator-mxnet/pull/13196
>
> Some of the examples failed after API changes and remained uncaught
> until
> a user reported the issue. While the community is actively working
> on
> fixing it, it might re-occur after few days if we don’t have a
> proper
> mechanism to catch regressions.
>
> So, I would like to propose to enable nightly/weekly tests for the
> examples similar to what we have for tutorials to catch any such
> regressions. The test could check only basic
> functionalities/working
> of
> the
> examples. It can run small examples completely whereas it can run
> long
> training examples for only few epochs.
>
> Any thoughts from the community? Any other suggestions for fixing
> the
> same?
>
> Regards,
> Ankit Khedia
>
>
>
>
>
> --
> Sandeep Krishnamurthy
>
>
>
Re: Nightly/Weekly tests for examples
Posted by "Khedia, Ankit" <kh...@amazon.com.INVALID>.
We can start small consolidating few examples which are there in both tutorials/examples folder.
One of such example is GradCAM implementation:
In tutorials - https://github.com/apache/incubator-mxnet/blob/master/docs/tutorials/vision/cnn_visualization.md
In examples- https://github.com/apache/incubator-mxnet/blob/master/example/cnn_visualization
The only difference is we have a python script in place of notebook in examples folder for this particular example.
Any thoughts/suggestions?
On the other hand, I agree with Sandeep that there should be some basic testing of examples and we can start with a small set of python examples to begin with.
—Ankit
On Nov 13, 2018, at 10:31 AM, Naveen Swamy <mn...@gmail.com>> wrote:
Aaron, IMO tutorials have a specific purpose, to introduce concepts and
APIs to the users and I think converting examples to tutorials would
overwhelm the users, we should carefully choose which examples we want to
turn into tutorials.
I agree that today examples are graveyard of untested code, my suggestion
is to add some testing to the example when you touch the example - at the
least to check the functionality. These can be run once a week.
On Tue, Nov 13, 2018 at 6:52 AM Aaron Markham <aa...@gmail.com>>
wrote:
I've been actively promoting moving examples to tutorials during reviews.
That way they fall under the testing umbrella and get added to the website.
Many times there's not really a great distinction as to why something is in
the examples folder, other than it's like a graveyard of untested sample
code.
I would suggest a starting strategy of when doing updates on examples, see
if with just a little more effort, ask yourself, can it be converted to a
tutorial?
The last thing CI needs is more flaky tutorial tests, so whatever is done
here should use the more robust approaches that are being discussed.
Cheers,
Aaron
On Mon, Nov 12, 2018, 16:24 sandeep krishnamurthy <
sandeep.krishna98@gmail.com<ma...@gmail.com> wrote:
Thanks, Ankit for bringing this up. @Anirudh - All the concerns you
raised
are very valid. Here are my thoughts:
1. There were several examples that were crashing or had compiler errors.
This is a very bad user experience. All example scripts should be at
least runnable!
2. While I agree examples are too diverse (python scripts, notebooks,
epochs, print statements etc..) We can always start small, we can start
with 5 examples. We can use this to streamline all examples to be python
scripts, print statements, with the main function invoker that can take
params like epoch, dataset etc.
3. We can start with running weekly tests to avoid too long nightly test
pipeline.
4. One possible issue can be on a few examples that depend on a large or
controlled dataset. I am not sure yet, how to solve this, but, we can
think.
Any suggestions?
Best,
Sandeep
On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya <an...@gmail.com>>
wrote:
Hi Ankit,
I have a few concerns about testing examples. Before writing tests for
examples,
- you will need to first decide what constitutes a test for an
example,
because examples are not API calls, which will have return
statements
and
the test can just call the API and assert for certain values. Just
testing
if an example is a compilable python script will not add much value
in
my
opinion.
- And testing for example output and results will require a re-write
of
many of the examples, because many of them currently just have print
statements as outputs and does not return any value as such. I am
not
sure
if it is worth the dev-effort.
- the current set of examples in the mxnet repo are very diverse -
some
are written as python notebooks, some are just python scripts with
paper
implementations, and some are just illustrations of certain mxnet
features.
I am curious to know how you will write tests for these things.
Looking forward to seeing the design of this test bed/framework.
Thanks
Anirudh Acharya
On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
<ma...@googlemail.com.invalid>> wrote:
Hello Ankit,
that's a great idea! Using the tutorial tests as reference is a great
starting point. If you are interested, please don't hesitate to
attend
the
Berlin user group in case you would like to discuss your first
thoughts
in-person before drafting a design.
-Marco
Am Fr., 9. Nov. 2018, 23:23 hat khedia.ankit@gmail.com<ma...@gmail.com> <
khedia.ankit@gmail.com<ma...@gmail.com>> geschrieben:
Hi MXNet community,
Recently, I and a few other contributors focussed on fixing
examples
in
our repository which were not working out of the box as expected.
https://github.com/apache/incubator-mxnet/issues/12800
https://github.com/apache/incubator-mxnet/issues/11895
https://github.com/apache/incubator-mxnet/pull/13196
Some of the examples failed after API changes and remained uncaught
until
a user reported the issue. While the community is actively working
on
fixing it, it might re-occur after few days if we don’t have a
proper
mechanism to catch regressions.
So, I would like to propose to enable nightly/weekly tests for the
examples similar to what we have for tutorials to catch any such
regressions. The test could check only basic
functionalities/working
of
the
examples. It can run small examples completely whereas it can run
long
training examples for only few epochs.
Any thoughts from the community? Any other suggestions for fixing
the
same?
Regards,
Ankit Khedia
--
Sandeep Krishnamurthy
Re: Nightly/Weekly tests for examples
Posted by Naveen Swamy <mn...@gmail.com>.
Aaron, IMO tutorials have a specific purpose, to introduce concepts and
APIs to the users and I think converting examples to tutorials would
overwhelm the users, we should carefully choose which examples we want to
turn into tutorials.
I agree that today examples are graveyard of untested code, my suggestion
is to add some testing to the example when you touch the example - at the
least to check the functionality. These can be run once a week.
On Tue, Nov 13, 2018 at 6:52 AM Aaron Markham <aa...@gmail.com>
wrote:
> I've been actively promoting moving examples to tutorials during reviews.
> That way they fall under the testing umbrella and get added to the website.
>
> Many times there's not really a great distinction as to why something is in
> the examples folder, other than it's like a graveyard of untested sample
> code.
>
> I would suggest a starting strategy of when doing updates on examples, see
> if with just a little more effort, ask yourself, can it be converted to a
> tutorial?
>
> The last thing CI needs is more flaky tutorial tests, so whatever is done
> here should use the more robust approaches that are being discussed.
>
> Cheers,
> Aaron
>
> On Mon, Nov 12, 2018, 16:24 sandeep krishnamurthy <
> sandeep.krishna98@gmail.com wrote:
>
> > Thanks, Ankit for bringing this up. @Anirudh - All the concerns you
> raised
> > are very valid. Here are my thoughts:
> > 1. There were several examples that were crashing or had compiler errors.
> > This is a very bad user experience. All example scripts should be at
> > least runnable!
> > 2. While I agree examples are too diverse (python scripts, notebooks,
> > epochs, print statements etc..) We can always start small, we can start
> > with 5 examples. We can use this to streamline all examples to be python
> > scripts, print statements, with the main function invoker that can take
> > params like epoch, dataset etc.
> > 3. We can start with running weekly tests to avoid too long nightly test
> > pipeline.
> > 4. One possible issue can be on a few examples that depend on a large or
> > controlled dataset. I am not sure yet, how to solve this, but, we can
> > think.
> >
> > Any suggestions?
> > Best,
> > Sandeep
> >
> >
> >
> > On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya <an...@gmail.com>
> > wrote:
> >
> > > Hi Ankit,
> > >
> > > I have a few concerns about testing examples. Before writing tests for
> > > examples,
> > >
> > > - you will need to first decide what constitutes a test for an
> > example,
> > > because examples are not API calls, which will have return
> statements
> > > and
> > > the test can just call the API and assert for certain values. Just
> > > testing
> > > if an example is a compilable python script will not add much value
> in
> > > my
> > > opinion.
> > > - And testing for example output and results will require a re-write
> > of
> > > many of the examples, because many of them currently just have print
> > > statements as outputs and does not return any value as such. I am
> not
> > > sure
> > > if it is worth the dev-effort.
> > > - the current set of examples in the mxnet repo are very diverse -
> > some
> > > are written as python notebooks, some are just python scripts with
> > paper
> > > implementations, and some are just illustrations of certain mxnet
> > > features.
> > > I am curious to know how you will write tests for these things.
> > >
> > >
> > > Looking forward to seeing the design of this test bed/framework.
> > >
> > >
> > > Thanks
> > > Anirudh Acharya
> > >
> > > On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
> > > <ma...@googlemail.com.invalid> wrote:
> > >
> > > > Hello Ankit,
> > > >
> > > > that's a great idea! Using the tutorial tests as reference is a great
> > > > starting point. If you are interested, please don't hesitate to
> attend
> > > the
> > > > Berlin user group in case you would like to discuss your first
> thoughts
> > > > in-person before drafting a design.
> > > >
> > > > -Marco
> > > >
> > > >
> > > > Am Fr., 9. Nov. 2018, 23:23 hat khedia.ankit@gmail.com <
> > > > khedia.ankit@gmail.com> geschrieben:
> > > >
> > > > > Hi MXNet community,
> > > > >
> > > > > Recently, I and a few other contributors focussed on fixing
> examples
> > in
> > > > > our repository which were not working out of the box as expected.
> > > > > https://github.com/apache/incubator-mxnet/issues/12800
> > > > > https://github.com/apache/incubator-mxnet/issues/11895
> > > > > https://github.com/apache/incubator-mxnet/pull/13196
> > > > >
> > > > > Some of the examples failed after API changes and remained uncaught
> > > until
> > > > > a user reported the issue. While the community is actively working
> on
> > > > > fixing it, it might re-occur after few days if we don’t have a
> proper
> > > > > mechanism to catch regressions.
> > > > >
> > > > > So, I would like to propose to enable nightly/weekly tests for the
> > > > > examples similar to what we have for tutorials to catch any such
> > > > > regressions. The test could check only basic
> functionalities/working
> > of
> > > > the
> > > > > examples. It can run small examples completely whereas it can run
> > long
> > > > > training examples for only few epochs.
> > > > >
> > > > > Any thoughts from the community? Any other suggestions for fixing
> the
> > > > same?
> > > > >
> > > > > Regards,
> > > > > Ankit Khedia
> > > > >
> > > >
> > >
> >
> >
> > --
> > Sandeep Krishnamurthy
> >
>
Re: Nightly/Weekly tests for examples
Posted by Aaron Markham <aa...@gmail.com>.
I've been actively promoting moving examples to tutorials during reviews.
That way they fall under the testing umbrella and get added to the website.
Many times there's not really a great distinction as to why something is in
the examples folder, other than it's like a graveyard of untested sample
code.
I would suggest a starting strategy of when doing updates on examples, see
if with just a little more effort, ask yourself, can it be converted to a
tutorial?
The last thing CI needs is more flaky tutorial tests, so whatever is done
here should use the more robust approaches that are being discussed.
Cheers,
Aaron
On Mon, Nov 12, 2018, 16:24 sandeep krishnamurthy <
sandeep.krishna98@gmail.com wrote:
> Thanks, Ankit for bringing this up. @Anirudh - All the concerns you raised
> are very valid. Here are my thoughts:
> 1. There were several examples that were crashing or had compiler errors.
> This is a very bad user experience. All example scripts should be at
> least runnable!
> 2. While I agree examples are too diverse (python scripts, notebooks,
> epochs, print statements etc..) We can always start small, we can start
> with 5 examples. We can use this to streamline all examples to be python
> scripts, print statements, with the main function invoker that can take
> params like epoch, dataset etc.
> 3. We can start with running weekly tests to avoid too long nightly test
> pipeline.
> 4. One possible issue can be on a few examples that depend on a large or
> controlled dataset. I am not sure yet, how to solve this, but, we can
> think.
>
> Any suggestions?
> Best,
> Sandeep
>
>
>
> On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya <an...@gmail.com>
> wrote:
>
> > Hi Ankit,
> >
> > I have a few concerns about testing examples. Before writing tests for
> > examples,
> >
> > - you will need to first decide what constitutes a test for an
> example,
> > because examples are not API calls, which will have return statements
> > and
> > the test can just call the API and assert for certain values. Just
> > testing
> > if an example is a compilable python script will not add much value in
> > my
> > opinion.
> > - And testing for example output and results will require a re-write
> of
> > many of the examples, because many of them currently just have print
> > statements as outputs and does not return any value as such. I am not
> > sure
> > if it is worth the dev-effort.
> > - the current set of examples in the mxnet repo are very diverse -
> some
> > are written as python notebooks, some are just python scripts with
> paper
> > implementations, and some are just illustrations of certain mxnet
> > features.
> > I am curious to know how you will write tests for these things.
> >
> >
> > Looking forward to seeing the design of this test bed/framework.
> >
> >
> > Thanks
> > Anirudh Acharya
> >
> > On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
> > <ma...@googlemail.com.invalid> wrote:
> >
> > > Hello Ankit,
> > >
> > > that's a great idea! Using the tutorial tests as reference is a great
> > > starting point. If you are interested, please don't hesitate to attend
> > the
> > > Berlin user group in case you would like to discuss your first thoughts
> > > in-person before drafting a design.
> > >
> > > -Marco
> > >
> > >
> > > Am Fr., 9. Nov. 2018, 23:23 hat khedia.ankit@gmail.com <
> > > khedia.ankit@gmail.com> geschrieben:
> > >
> > > > Hi MXNet community,
> > > >
> > > > Recently, I and a few other contributors focussed on fixing examples
> in
> > > > our repository which were not working out of the box as expected.
> > > > https://github.com/apache/incubator-mxnet/issues/12800
> > > > https://github.com/apache/incubator-mxnet/issues/11895
> > > > https://github.com/apache/incubator-mxnet/pull/13196
> > > >
> > > > Some of the examples failed after API changes and remained uncaught
> > until
> > > > a user reported the issue. While the community is actively working on
> > > > fixing it, it might re-occur after few days if we don’t have a proper
> > > > mechanism to catch regressions.
> > > >
> > > > So, I would like to propose to enable nightly/weekly tests for the
> > > > examples similar to what we have for tutorials to catch any such
> > > > regressions. The test could check only basic functionalities/working
> of
> > > the
> > > > examples. It can run small examples completely whereas it can run
> long
> > > > training examples for only few epochs.
> > > >
> > > > Any thoughts from the community? Any other suggestions for fixing the
> > > same?
> > > >
> > > > Regards,
> > > > Ankit Khedia
> > > >
> > >
> >
>
>
> --
> Sandeep Krishnamurthy
>
Re: Nightly/Weekly tests for examples
Posted by sandeep krishnamurthy <sa...@gmail.com>.
Thanks, Ankit for bringing this up. @Anirudh - All the concerns you raised
are very valid. Here are my thoughts:
1. There were several examples that were crashing or had compiler errors.
This is a very bad user experience. All example scripts should be at
least runnable!
2. While I agree examples are too diverse (python scripts, notebooks,
epochs, print statements etc..) We can always start small, we can start
with 5 examples. We can use this to streamline all examples to be python
scripts, print statements, with the main function invoker that can take
params like epoch, dataset etc.
3. We can start with running weekly tests to avoid too long nightly test
pipeline.
4. One possible issue can be on a few examples that depend on a large or
controlled dataset. I am not sure yet, how to solve this, but, we can think.
Any suggestions?
Best,
Sandeep
On Mon, Nov 12, 2018 at 10:38 AM Anirudh Acharya <an...@gmail.com>
wrote:
> Hi Ankit,
>
> I have a few concerns about testing examples. Before writing tests for
> examples,
>
> - you will need to first decide what constitutes a test for an example,
> because examples are not API calls, which will have return statements
> and
> the test can just call the API and assert for certain values. Just
> testing
> if an example is a compilable python script will not add much value in
> my
> opinion.
> - And testing for example output and results will require a re-write of
> many of the examples, because many of them currently just have print
> statements as outputs and does not return any value as such. I am not
> sure
> if it is worth the dev-effort.
> - the current set of examples in the mxnet repo are very diverse - some
> are written as python notebooks, some are just python scripts with paper
> implementations, and some are just illustrations of certain mxnet
> features.
> I am curious to know how you will write tests for these things.
>
>
> Looking forward to seeing the design of this test bed/framework.
>
>
> Thanks
> Anirudh Acharya
>
> On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
> <ma...@googlemail.com.invalid> wrote:
>
> > Hello Ankit,
> >
> > that's a great idea! Using the tutorial tests as reference is a great
> > starting point. If you are interested, please don't hesitate to attend
> the
> > Berlin user group in case you would like to discuss your first thoughts
> > in-person before drafting a design.
> >
> > -Marco
> >
> >
> > Am Fr., 9. Nov. 2018, 23:23 hat khedia.ankit@gmail.com <
> > khedia.ankit@gmail.com> geschrieben:
> >
> > > Hi MXNet community,
> > >
> > > Recently, I and a few other contributors focussed on fixing examples in
> > > our repository which were not working out of the box as expected.
> > > https://github.com/apache/incubator-mxnet/issues/12800
> > > https://github.com/apache/incubator-mxnet/issues/11895
> > > https://github.com/apache/incubator-mxnet/pull/13196
> > >
> > > Some of the examples failed after API changes and remained uncaught
> until
> > > a user reported the issue. While the community is actively working on
> > > fixing it, it might re-occur after few days if we don’t have a proper
> > > mechanism to catch regressions.
> > >
> > > So, I would like to propose to enable nightly/weekly tests for the
> > > examples similar to what we have for tutorials to catch any such
> > > regressions. The test could check only basic functionalities/working of
> > the
> > > examples. It can run small examples completely whereas it can run long
> > > training examples for only few epochs.
> > >
> > > Any thoughts from the community? Any other suggestions for fixing the
> > same?
> > >
> > > Regards,
> > > Ankit Khedia
> > >
> >
>
--
Sandeep Krishnamurthy
Re: Nightly/Weekly tests for examples
Posted by Anirudh Acharya <an...@gmail.com>.
Hi Ankit,
I have a few concerns about testing examples. Before writing tests for
examples,
- you will need to first decide what constitutes a test for an example,
because examples are not API calls, which will have return statements and
the test can just call the API and assert for certain values. Just testing
if an example is a compilable python script will not add much value in my
opinion.
- And testing for example output and results will require a re-write of
many of the examples, because many of them currently just have print
statements as outputs and does not return any value as such. I am not sure
if it is worth the dev-effort.
- the current set of examples in the mxnet repo are very diverse - some
are written as python notebooks, some are just python scripts with paper
implementations, and some are just illustrations of certain mxnet features.
I am curious to know how you will write tests for these things.
Looking forward to seeing the design of this test bed/framework.
Thanks
Anirudh Acharya
On Fri, Nov 9, 2018 at 2:39 PM Marco de Abreu
<ma...@googlemail.com.invalid> wrote:
> Hello Ankit,
>
> that's a great idea! Using the tutorial tests as reference is a great
> starting point. If you are interested, please don't hesitate to attend the
> Berlin user group in case you would like to discuss your first thoughts
> in-person before drafting a design.
>
> -Marco
>
>
> Am Fr., 9. Nov. 2018, 23:23 hat khedia.ankit@gmail.com <
> khedia.ankit@gmail.com> geschrieben:
>
> > Hi MXNet community,
> >
> > Recently, I and a few other contributors focussed on fixing examples in
> > our repository which were not working out of the box as expected.
> > https://github.com/apache/incubator-mxnet/issues/12800
> > https://github.com/apache/incubator-mxnet/issues/11895
> > https://github.com/apache/incubator-mxnet/pull/13196
> >
> > Some of the examples failed after API changes and remained uncaught until
> > a user reported the issue. While the community is actively working on
> > fixing it, it might re-occur after few days if we don’t have a proper
> > mechanism to catch regressions.
> >
> > So, I would like to propose to enable nightly/weekly tests for the
> > examples similar to what we have for tutorials to catch any such
> > regressions. The test could check only basic functionalities/working of
> the
> > examples. It can run small examples completely whereas it can run long
> > training examples for only few epochs.
> >
> > Any thoughts from the community? Any other suggestions for fixing the
> same?
> >
> > Regards,
> > Ankit Khedia
> >
>
Re: Nightly/Weekly tests for examples
Posted by Marco de Abreu <ma...@googlemail.com.INVALID>.
Hello Ankit,
that's a great idea! Using the tutorial tests as reference is a great
starting point. If you are interested, please don't hesitate to attend the
Berlin user group in case you would like to discuss your first thoughts
in-person before drafting a design.
-Marco
Am Fr., 9. Nov. 2018, 23:23 hat khedia.ankit@gmail.com <
khedia.ankit@gmail.com> geschrieben:
> Hi MXNet community,
>
> Recently, I and a few other contributors focussed on fixing examples in
> our repository which were not working out of the box as expected.
> https://github.com/apache/incubator-mxnet/issues/12800
> https://github.com/apache/incubator-mxnet/issues/11895
> https://github.com/apache/incubator-mxnet/pull/13196
>
> Some of the examples failed after API changes and remained uncaught until
> a user reported the issue. While the community is actively working on
> fixing it, it might re-occur after few days if we don’t have a proper
> mechanism to catch regressions.
>
> So, I would like to propose to enable nightly/weekly tests for the
> examples similar to what we have for tutorials to catch any such
> regressions. The test could check only basic functionalities/working of the
> examples. It can run small examples completely whereas it can run long
> training examples for only few epochs.
>
> Any thoughts from the community? Any other suggestions for fixing the same?
>
> Regards,
> Ankit Khedia
>