Testing examples in nightly build

[Vandana Kannan <va...@gmail.com>]

Hi All,

Recently we saw that there were quite a few Pylint undefined-variable errors in MXNet code, a majority of them in the example folder. These errors highlight that there are paths in the examples that are broken and these have not been caught during testing. 

These errors are later reported as issues by users. For example,
https://github.com/apache/incubator-mxnet/issues/11278.

It would be a good idea to include tests for all examples in the nightly builds (Similar to incubator-mxnet/tests/nightly/test_image_classification.sh). Side note: We might have to see what to do about examples that include special setup instructions or take too long to execute.

Any thoughts or suggestions on writing these tests, feasibility or previous attempts at including these tests, would be helpful.

Thanks,
Vandana

Re: Testing examples in nightly build

[kellen sunderland <ke...@gmail.com>]

@Vandana: Yes the backlog would need to be addressed, but I've been very
encouraged by your and cclauss's recent work in this area.  Thanks a ton
for the effort, I think it's really improving the quality.

On Thu, Aug 16, 2018 at 8:15 PM Marco de Abreu
<ma...@googlemail.com.invalid> wrote:

> These are great ideas, sounds very good to me. Thanks for your efforts
> around the user experience!
>
> -Marco
>
> Vandana Kannan <va...@gmail.com> schrieb am Do., 16. Aug. 2018,
> 18:27:
>
> > @Marco: Most of the examples are standalone applications and not
> > notebooks. We could start off by enabling tests for examples that are
> > either notebooks or those that can be executed with command line options.
> >
> > Some of the problems that we may come across are with examples that load
> > large datasets (we could probably use a smaller dataset, but it depends
> on
> > the example), or examples that have a long training time (we could
> probably
> > execute them for 1 epoch). Straight Dope nightly tests cap the execution
> > time to 10 minutes per notebook. We could do the same here wherever
> > possible.
> >
> > @Kellen: Good idea to include flake8/pylint checks for examples in
> nightly
> > builds - we would need to fix the backlog (
> > https://github.com/apache/incubator-mxnet/issues/12205) before enabling.
> > Yes, any contribution from the community on any of these TODO items,
> would
> > be great.
> >
> > - Vandana
> >
> > On 2018/08/16 06:40:56, kellen sunderland <ke...@gmail.com>
> > wrote:
> > > I think it would be very beneficial to start fleshing out the nighties.
> > > They provide a lot of value at a relatively small cost.  Any
> > contributions
> > > from the community would be appreciated.
> > >
> > > Things I could see being beneficial:
> > > * Long running tests
> > > * In depth flake8/pylint linters
> > > * cudamemcheck builds
> > > * Asan builds.
> > >
> > > On Thu, Aug 16, 2018, 12:24 AM Marco de Abreu
> > > <ma...@googlemail.com.invalid> wrote:
> > >
> > > > Hello,
> > > >
> > > > I think this is a great idea! Thanks a lot for fixing all the
> problems
> > in
> > > > our examples.
> > > >
> > > > Do you see any problems that could come up if we just run them one by
> > one?
> > > > We already got a pipeline that allows us to verify jupyter notebooks,
> > but
> > > > afaik the examples are standalone files with a main function, right?
> > > >
> > > > Best regards,
> > > > Marco
> > > >
> > > > Vandana Kannan <va...@gmail.com> schrieb am Mi., 15. Aug. 2018,
> > > > 23:44:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Recently we saw that there were quite a few Pylint
> undefined-variable
> > > > > errors in MXNet code, a majority of them in the example folder.
> These
> > > > > errors highlight that there are paths in the examples that are
> > broken and
> > > > > these have not been caught during testing.
> > > > >
> > > > > These errors are later reported as issues by users. For example,
> > > > > https://github.com/apache/incubator-mxnet/issues/11278.
> > > > >
> > > > > It would be a good idea to include tests for all examples in the
> > nightly
> > > > > builds (Similar to
> > > > > incubator-mxnet/tests/nightly/test_image_classification.sh). Side
> > note:
> > > > We
> > > > > might have to see what to do about examples that include special
> > setup
> > > > > instructions or take too long to execute.
> > > > >
> > > > > Any thoughts or suggestions on writing these tests, feasibility or
> > > > > previous attempts at including these tests, would be helpful.
> > > > >
> > > > > Thanks,
> > > > > Vandana
> > > > >
> > > >
> > >
> >
> >
>

Re: Testing examples in nightly build

[Marco de Abreu <ma...@googlemail.com.INVALID>]

These are great ideas, sounds very good to me. Thanks for your efforts
around the user experience!

-Marco

Vandana Kannan <va...@gmail.com> schrieb am Do., 16. Aug. 2018, 18:27:

> @Marco: Most of the examples are standalone applications and not
> notebooks. We could start off by enabling tests for examples that are
> either notebooks or those that can be executed with command line options.
>
> Some of the problems that we may come across are with examples that load
> large datasets (we could probably use a smaller dataset, but it depends on
> the example), or examples that have a long training time (we could probably
> execute them for 1 epoch). Straight Dope nightly tests cap the execution
> time to 10 minutes per notebook. We could do the same here wherever
> possible.
>
> @Kellen: Good idea to include flake8/pylint checks for examples in nightly
> builds - we would need to fix the backlog (
> https://github.com/apache/incubator-mxnet/issues/12205) before enabling.
> Yes, any contribution from the community on any of these TODO items, would
> be great.
>
> - Vandana
>
> On 2018/08/16 06:40:56, kellen sunderland <ke...@gmail.com>
> wrote:
> > I think it would be very beneficial to start fleshing out the nighties.
> > They provide a lot of value at a relatively small cost.  Any
> contributions
> > from the community would be appreciated.
> >
> > Things I could see being beneficial:
> > * Long running tests
> > * In depth flake8/pylint linters
> > * cudamemcheck builds
> > * Asan builds.
> >
> > On Thu, Aug 16, 2018, 12:24 AM Marco de Abreu
> > <ma...@googlemail.com.invalid> wrote:
> >
> > > Hello,
> > >
> > > I think this is a great idea! Thanks a lot for fixing all the problems
> in
> > > our examples.
> > >
> > > Do you see any problems that could come up if we just run them one by
> one?
> > > We already got a pipeline that allows us to verify jupyter notebooks,
> but
> > > afaik the examples are standalone files with a main function, right?
> > >
> > > Best regards,
> > > Marco
> > >
> > > Vandana Kannan <va...@gmail.com> schrieb am Mi., 15. Aug. 2018,
> > > 23:44:
> > >
> > > > Hi All,
> > > >
> > > > Recently we saw that there were quite a few Pylint undefined-variable
> > > > errors in MXNet code, a majority of them in the example folder. These
> > > > errors highlight that there are paths in the examples that are
> broken and
> > > > these have not been caught during testing.
> > > >
> > > > These errors are later reported as issues by users. For example,
> > > > https://github.com/apache/incubator-mxnet/issues/11278.
> > > >
> > > > It would be a good idea to include tests for all examples in the
> nightly
> > > > builds (Similar to
> > > > incubator-mxnet/tests/nightly/test_image_classification.sh). Side
> note:
> > > We
> > > > might have to see what to do about examples that include special
> setup
> > > > instructions or take too long to execute.
> > > >
> > > > Any thoughts or suggestions on writing these tests, feasibility or
> > > > previous attempts at including these tests, would be helpful.
> > > >
> > > > Thanks,
> > > > Vandana
> > > >
> > >
> >
>
>

Re: Testing examples in nightly build

[Vandana Kannan <va...@gmail.com>]

@Marco: Most of the examples are standalone applications and not notebooks. We could start off by enabling tests for examples that are either notebooks or those that can be executed with command line options.

Some of the problems that we may come across are with examples that load large datasets (we could probably use a smaller dataset, but it depends on the example), or examples that have a long training time (we could probably execute them for 1 epoch). Straight Dope nightly tests cap the execution time to 10 minutes per notebook. We could do the same here wherever possible. 

@Kellen: Good idea to include flake8/pylint checks for examples in nightly builds - we would need to fix the backlog (https://github.com/apache/incubator-mxnet/issues/12205) before enabling. Yes, any contribution from the community on any of these TODO items, would be great.

- Vandana

On 2018/08/16 06:40:56, kellen sunderland <ke...@gmail.com> wrote: 
> I think it would be very beneficial to start fleshing out the nighties.
> They provide a lot of value at a relatively small cost.  Any contributions
> from the community would be appreciated.
> 
> Things I could see being beneficial:
> * Long running tests
> * In depth flake8/pylint linters
> * cudamemcheck builds
> * Asan builds.
> 
> On Thu, Aug 16, 2018, 12:24 AM Marco de Abreu
> <ma...@googlemail.com.invalid> wrote:
> 
> > Hello,
> >
> > I think this is a great idea! Thanks a lot for fixing all the problems in
> > our examples.
> >
> > Do you see any problems that could come up if we just run them one by one?
> > We already got a pipeline that allows us to verify jupyter notebooks, but
> > afaik the examples are standalone files with a main function, right?
> >
> > Best regards,
> > Marco
> >
> > Vandana Kannan <va...@gmail.com> schrieb am Mi., 15. Aug. 2018,
> > 23:44:
> >
> > > Hi All,
> > >
> > > Recently we saw that there were quite a few Pylint undefined-variable
> > > errors in MXNet code, a majority of them in the example folder. These
> > > errors highlight that there are paths in the examples that are broken and
> > > these have not been caught during testing.
> > >
> > > These errors are later reported as issues by users. For example,
> > > https://github.com/apache/incubator-mxnet/issues/11278.
> > >
> > > It would be a good idea to include tests for all examples in the nightly
> > > builds (Similar to
> > > incubator-mxnet/tests/nightly/test_image_classification.sh). Side note:
> > We
> > > might have to see what to do about examples that include special setup
> > > instructions or take too long to execute.
> > >
> > > Any thoughts or suggestions on writing these tests, feasibility or
> > > previous attempts at including these tests, would be helpful.
> > >
> > > Thanks,
> > > Vandana
> > >
> >
> 

Re: Testing examples in nightly build

[kellen sunderland <ke...@gmail.com>]

I think it would be very beneficial to start fleshing out the nighties.
They provide a lot of value at a relatively small cost.  Any contributions
from the community would be appreciated.

Things I could see being beneficial:
* Long running tests
* In depth flake8/pylint linters
* cudamemcheck builds
* Asan builds.

On Thu, Aug 16, 2018, 12:24 AM Marco de Abreu
<ma...@googlemail.com.invalid> wrote:

> Hello,
>
> I think this is a great idea! Thanks a lot for fixing all the problems in
> our examples.
>
> Do you see any problems that could come up if we just run them one by one?
> We already got a pipeline that allows us to verify jupyter notebooks, but
> afaik the examples are standalone files with a main function, right?
>
> Best regards,
> Marco
>
> Vandana Kannan <va...@gmail.com> schrieb am Mi., 15. Aug. 2018,
> 23:44:
>
> > Hi All,
> >
> > Recently we saw that there were quite a few Pylint undefined-variable
> > errors in MXNet code, a majority of them in the example folder. These
> > errors highlight that there are paths in the examples that are broken and
> > these have not been caught during testing.
> >
> > These errors are later reported as issues by users. For example,
> > https://github.com/apache/incubator-mxnet/issues/11278.
> >
> > It would be a good idea to include tests for all examples in the nightly
> > builds (Similar to
> > incubator-mxnet/tests/nightly/test_image_classification.sh). Side note:
> We
> > might have to see what to do about examples that include special setup
> > instructions or take too long to execute.
> >
> > Any thoughts or suggestions on writing these tests, feasibility or
> > previous attempts at including these tests, would be helpful.
> >
> > Thanks,
> > Vandana
> >
>

Re: Testing examples in nightly build

[Marco de Abreu <ma...@googlemail.com.INVALID>]

Hello,

I think this is a great idea! Thanks a lot for fixing all the problems in
our examples.

Do you see any problems that could come up if we just run them one by one?
We already got a pipeline that allows us to verify jupyter notebooks, but
afaik the examples are standalone files with a main function, right?

Best regards,
Marco

Vandana Kannan <va...@gmail.com> schrieb am Mi., 15. Aug. 2018, 23:44:

> Hi All,
>
> Recently we saw that there were quite a few Pylint undefined-variable
> errors in MXNet code, a majority of them in the example folder. These
> errors highlight that there are paths in the examples that are broken and
> these have not been caught during testing.
>
> These errors are later reported as issues by users. For example,
> https://github.com/apache/incubator-mxnet/issues/11278.
>
> It would be a good idea to include tests for all examples in the nightly
> builds (Similar to
> incubator-mxnet/tests/nightly/test_image_classification.sh). Side note: We
> might have to see what to do about examples that include special setup
> instructions or take too long to execute.
>
> Any thoughts or suggestions on writing these tests, feasibility or
> previous attempts at including these tests, would be helpful.
>
> Thanks,
> Vandana
>