You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by Qing Lan <la...@live.com> on 2018/07/11 22:47:44 UTC

Proposal to release the Examples for Scala in V1.3

Hi all,

I would like to propose the release plan for MXNet Scala examples in V1.3. Currently, the examples in the main repository are not maintained for a long time. I have spent some time to use the improved Scala API (known as the .api) to improve the examples with documentation to run and add CI tests. However, I am facing the difficulties to resolve memory leaks as most of the examples did not contains memory leak fixes. It crashes the CI because of that. We can temporarily disable the CI test for some of the examples and release them with 1.3 and enable them once we have a better GC in MXNet Scala. Here is a brief Pros and Cons for this release:

Pros:

  *   A good set of demos on how to use improved Training API
  *   Examples are fixed with full documentation to help to run them

Cons:

  *   Examples contains the memory leak issues may cause problems

Please kindly share your ideas in this thread and I am really appreciate for your help.

Thanks,
Qing

Re: Proposal to release the Examples for Scala in V1.3

Posted by Qing Lan <la...@live.com>.
Hi Marco,

Currently all Scala examples I have changed contains test coverage on CI (in the example/src/test/). It's been temporarily disabled because of the memory leaks.

I would also like to raise a discussion on a general standard for testing examples, this is what Scala side doing:

1. For small training examples we can finish them and check the dev accuracy
2. For big training examples, we do 5-10 Epoch to check it is runnable
3. For trivial examples (like neural style), we check it is runnable
4. For inference examples and classification specifically, we run with several test cases to measure if it can get the result correctly.

Thanks,
Qing

On 7/12/18, 11:00 AM, "Marco de Abreu" <ma...@googlemail.com.INVALID> wrote:

    Hi Qing,
    
    thanks a lot for your efforts around the Scala examples and to assist us
    getting the Scala API to a broader user base! This is a great user-facing
    approach!
    
    The following might not be in the current course of how we handle examples
    in Python, but let me elaborate: I have seen a lot of problems in the last
    months related to examples in Python in the last months that could have
    been avoided by testing them in integration tests. These errors included
    syntax errors, typos, wrong logic, outdated APIs or other more
    context-related errors. Thus, I'd like to strive for full test coverage of
    our examples in the long term. I know that we can't get this done within a
    single night, but since you are now reworking the way our examples work in
    Scala, I'd like to kindly request to assess whether it's possible to
    include testing of Scala examples in your plan right from the start. This
    will ensure that we have a good first impression with our users, increases
    our overall test coverage and also reduces the maintenance overhead we are
    having with examples breaking without us knowing.
    
    I think the memory leaks are adjacent problems and that examples are a good
    way to make them visible. Since they are already there, we shouldn't block
    ourselves from writing and publishing examples. Our users would encounter
    them anyways, but by making the examples, we can properly document and
    track them ahead of time. The big advantage right now would be that we
    could mark these things as known problems in our patch notes and thus our
    users would know that we are aware of them and that we are going to fix
    them by (hopefully) the following release.
    
    Best regards,
    Marco
    
    On Thu, Jul 12, 2018 at 8:41 PM YiZhi Liu <ea...@gmail.com> wrote:
    
    > Agree to make incremental improvements. As long as the changes do not
    > change the current states of memory leak, but improve the documents
    > and demonstrate the way to use type-safe apis, I think it is fair to
    > merge into v1.3.
    > On Wed, Jul 11, 2018 at 4:43 PM Naveen Swamy <mn...@gmail.com> wrote:
    > >
    > > Qing, its great that you are working on improving the Scala Examples with
    > > documentation and tests.
    > > Ideally, we shouldn't have any memory leaks..it erodes trust with our
    > > users, however I understand it can take significant time and debugging
    > > effort to resolve them. Given that these leaks already exist, may be
    > should
    > > call out on the documentation that this is a known issue and we will work
    > > on it.
    > > IMO for this case making incremental improvements to examples makes
    > sense.
    > >
    > > I would like to hear what others think.
    > > -Naveen
    > >
    > > On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan <la...@live.com> wrote:
    > >
    > > > Hi all,
    > > >
    > > > I would like to propose the release plan for MXNet Scala examples in
    > V1.3.
    > > > Currently, the examples in the main repository are not maintained for a
    > > > long time. I have spent some time to use the improved Scala API (known
    > as
    > > > the .api) to improve the examples with documentation to run and add CI
    > > > tests. However, I am facing the difficulties to resolve memory leaks as
    > > > most of the examples did not contains memory leak fixes. It crashes
    > the CI
    > > > because of that. We can temporarily disable the CI test for some of the
    > > > examples and release them with 1.3 and enable them once we have a
    > better GC
    > > > in MXNet Scala. Here is a brief Pros and Cons for this release:
    > > >
    > > > Pros:
    > > >
    > > >   *   A good set of demos on how to use improved Training API
    > > >   *   Examples are fixed with full documentation to help to run them
    > > >
    > > > Cons:
    > > >
    > > >   *   Examples contains the memory leak issues may cause problems
    > > >
    > > > Please kindly share your ideas in this thread and I am really
    > appreciate
    > > > for your help.
    > > >
    > > > Thanks,
    > > > Qing
    > > >
    >
    >
    >
    > --
    > Yizhi Liu
    > DMLC member
    > Amazon Web Services
    > Vancouver, Canada
    >

Re: Proposal to release the Examples for Scala in V1.3

Posted by Marco de Abreu <ma...@googlemail.com.INVALID>.
Hi Qing,

thanks a lot for your efforts around the Scala examples and to assist us
getting the Scala API to a broader user base! This is a great user-facing
approach!

The following might not be in the current course of how we handle examples
in Python, but let me elaborate: I have seen a lot of problems in the last
months related to examples in Python in the last months that could have
been avoided by testing them in integration tests. These errors included
syntax errors, typos, wrong logic, outdated APIs or other more
context-related errors. Thus, I'd like to strive for full test coverage of
our examples in the long term. I know that we can't get this done within a
single night, but since you are now reworking the way our examples work in
Scala, I'd like to kindly request to assess whether it's possible to
include testing of Scala examples in your plan right from the start. This
will ensure that we have a good first impression with our users, increases
our overall test coverage and also reduces the maintenance overhead we are
having with examples breaking without us knowing.

I think the memory leaks are adjacent problems and that examples are a good
way to make them visible. Since they are already there, we shouldn't block
ourselves from writing and publishing examples. Our users would encounter
them anyways, but by making the examples, we can properly document and
track them ahead of time. The big advantage right now would be that we
could mark these things as known problems in our patch notes and thus our
users would know that we are aware of them and that we are going to fix
them by (hopefully) the following release.

Best regards,
Marco

On Thu, Jul 12, 2018 at 8:41 PM YiZhi Liu <ea...@gmail.com> wrote:

> Agree to make incremental improvements. As long as the changes do not
> change the current states of memory leak, but improve the documents
> and demonstrate the way to use type-safe apis, I think it is fair to
> merge into v1.3.
> On Wed, Jul 11, 2018 at 4:43 PM Naveen Swamy <mn...@gmail.com> wrote:
> >
> > Qing, its great that you are working on improving the Scala Examples with
> > documentation and tests.
> > Ideally, we shouldn't have any memory leaks..it erodes trust with our
> > users, however I understand it can take significant time and debugging
> > effort to resolve them. Given that these leaks already exist, may be
> should
> > call out on the documentation that this is a known issue and we will work
> > on it.
> > IMO for this case making incremental improvements to examples makes
> sense.
> >
> > I would like to hear what others think.
> > -Naveen
> >
> > On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan <la...@live.com> wrote:
> >
> > > Hi all,
> > >
> > > I would like to propose the release plan for MXNet Scala examples in
> V1.3.
> > > Currently, the examples in the main repository are not maintained for a
> > > long time. I have spent some time to use the improved Scala API (known
> as
> > > the .api) to improve the examples with documentation to run and add CI
> > > tests. However, I am facing the difficulties to resolve memory leaks as
> > > most of the examples did not contains memory leak fixes. It crashes
> the CI
> > > because of that. We can temporarily disable the CI test for some of the
> > > examples and release them with 1.3 and enable them once we have a
> better GC
> > > in MXNet Scala. Here is a brief Pros and Cons for this release:
> > >
> > > Pros:
> > >
> > >   *   A good set of demos on how to use improved Training API
> > >   *   Examples are fixed with full documentation to help to run them
> > >
> > > Cons:
> > >
> > >   *   Examples contains the memory leak issues may cause problems
> > >
> > > Please kindly share your ideas in this thread and I am really
> appreciate
> > > for your help.
> > >
> > > Thanks,
> > > Qing
> > >
>
>
>
> --
> Yizhi Liu
> DMLC member
> Amazon Web Services
> Vancouver, Canada
>

Re: Proposal to release the Examples for Scala in V1.3

Posted by YiZhi Liu <ea...@gmail.com>.
Agree to make incremental improvements. As long as the changes do not
change the current states of memory leak, but improve the documents
and demonstrate the way to use type-safe apis, I think it is fair to
merge into v1.3.
On Wed, Jul 11, 2018 at 4:43 PM Naveen Swamy <mn...@gmail.com> wrote:
>
> Qing, its great that you are working on improving the Scala Examples with
> documentation and tests.
> Ideally, we shouldn't have any memory leaks..it erodes trust with our
> users, however I understand it can take significant time and debugging
> effort to resolve them. Given that these leaks already exist, may be should
> call out on the documentation that this is a known issue and we will work
> on it.
> IMO for this case making incremental improvements to examples makes sense.
>
> I would like to hear what others think.
> -Naveen
>
> On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan <la...@live.com> wrote:
>
> > Hi all,
> >
> > I would like to propose the release plan for MXNet Scala examples in V1.3.
> > Currently, the examples in the main repository are not maintained for a
> > long time. I have spent some time to use the improved Scala API (known as
> > the .api) to improve the examples with documentation to run and add CI
> > tests. However, I am facing the difficulties to resolve memory leaks as
> > most of the examples did not contains memory leak fixes. It crashes the CI
> > because of that. We can temporarily disable the CI test for some of the
> > examples and release them with 1.3 and enable them once we have a better GC
> > in MXNet Scala. Here is a brief Pros and Cons for this release:
> >
> > Pros:
> >
> >   *   A good set of demos on how to use improved Training API
> >   *   Examples are fixed with full documentation to help to run them
> >
> > Cons:
> >
> >   *   Examples contains the memory leak issues may cause problems
> >
> > Please kindly share your ideas in this thread and I am really appreciate
> > for your help.
> >
> > Thanks,
> > Qing
> >



-- 
Yizhi Liu
DMLC member
Amazon Web Services
Vancouver, Canada

Re: Proposal to release the Examples for Scala in V1.3

Posted by Naveen Swamy <mn...@gmail.com>.
Qing, its great that you are working on improving the Scala Examples with
documentation and tests.
Ideally, we shouldn't have any memory leaks..it erodes trust with our
users, however I understand it can take significant time and debugging
effort to resolve them. Given that these leaks already exist, may be should
call out on the documentation that this is a known issue and we will work
on it.
IMO for this case making incremental improvements to examples makes sense.

I would like to hear what others think.
-Naveen

On Wed, Jul 11, 2018 at 3:47 PM, Qing Lan <la...@live.com> wrote:

> Hi all,
>
> I would like to propose the release plan for MXNet Scala examples in V1.3.
> Currently, the examples in the main repository are not maintained for a
> long time. I have spent some time to use the improved Scala API (known as
> the .api) to improve the examples with documentation to run and add CI
> tests. However, I am facing the difficulties to resolve memory leaks as
> most of the examples did not contains memory leak fixes. It crashes the CI
> because of that. We can temporarily disable the CI test for some of the
> examples and release them with 1.3 and enable them once we have a better GC
> in MXNet Scala. Here is a brief Pros and Cons for this release:
>
> Pros:
>
>   *   A good set of demos on how to use improved Training API
>   *   Examples are fixed with full documentation to help to run them
>
> Cons:
>
>   *   Examples contains the memory leak issues may cause problems
>
> Please kindly share your ideas in this thread and I am really appreciate
> for your help.
>
> Thanks,
> Qing
>