You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@mxnet.apache.org by Qing Lan <la...@live.com> on 2018/07/18 22:43:18 UTC

Data Type improvement in MXNet Scala

Hi all,

I would like to propose a change on the MXNet Scala data_type:

Depreciate the `provideData` and `provideLabel` functions to provide `DataDesc` type instead of `ListMap[String, Shape]` type.

There were several issues regarding to this since v0.12 on Scala. Several users reported that the `implicit def ListMap2Descs` helps to do some downcasting job however missing information during this process (layout property).
MXNet Python language has already change this part into `DataDesc`. In that case, I plan to add two more methods that will take `DataDesc` and depreciate the functions that mentioned above. The downcasting method `ListMap2Descs` will also be depreciated and not to be used in current MXNet Scala code base.

Thanks,
Qing

References:
Code position:
https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L312-L315
https://github.com/apache/incubator-mxnet/blob/master/scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L358-L363

Issue links:
https://github.com/apache/incubator-mxnet/issues/11775
https://github.com/apache/incubator-mxnet/issues/10753
https://github.com/apache/incubator-mxnet/issues/11571

Re: Data Type improvement in MXNet Scala

Posted by Qing Lan <la...@live.com>.

Thanks Naveen and Sandeep for your comments.

About the RNN model, the RNNBucketing class also meet the same issue that I have mentioned.
The change will be applied on the dataIter only. For example , NCHW will be mapped into a shape with size 4:
Shape:(0, 1, 2, 3)
Layout:(N,C,H,W)
So I think it is position-based.

Thanks,
Qing

On 7/18/18, 10:00 PM, "sandeep krishnamurthy" <sa...@gmail.com> wrote:

    Hi Qing,
    Thanks for picking this issue.
    How would multiple inputs/labels be handled? By name or by position?
    
    Background:
    Recently I faced issue in Python where name of the input symbol name was
    not used, but, position was used. Python2 reads list from 0 and Python3
    reads list from last element. So based on the python version, I was having
    swapped symbol and hence shape mis match errors.
    
    Best
    Sandeep
    
    On Wed, Jul 18, 2018 at 8:09 PM Naveen Swamy <mn...@gmail.com> wrote:
    
    > Thanks for looking into this Qing :)
    >
    > this would alleviate the problems we have in getting RNNs to work in Scala.
    > Can we call name the new APIs `provideDataDesc` and `provideLabelDesc` ?
    >
    > Thanks, Naveen
    >
    > On Wed, Jul 18, 2018 at 3:43 PM, Qing Lan <la...@live.com> wrote:
    >
    > > Hi all,
    > >
    > > I would like to propose a change on the MXNet Scala data_type:
    > >
    > > Depreciate the `provideData` and `provideLabel` functions to provide
    > > `DataDesc` type instead of `ListMap[String, Shape]` type.
    > >
    > > There were several issues regarding to this since v0.12 on Scala. Several
    > > users reported that the `implicit def ListMap2Descs` helps to do some
    > > downcasting job however missing information during this process (layout
    > > property).
    > > MXNet Python language has already change this part into `DataDesc`. In
    > > that case, I plan to add two more methods that will take `DataDesc` and
    > > depreciate the functions that mentioned above. The downcasting method
    > > `ListMap2Descs` will also be depreciated and not to be used in current
    > > MXNet Scala code base.
    > >
    > > Thanks,
    > > Qing
    > >
    > > References:
    > > Code position:
    > > https://github.com/apache/incubator-mxnet/blob/master/
    > > scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L312-L315
    > > https://github.com/apache/incubator-mxnet/blob/master/
    > > scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L358-L363
    > >
    > > Issue links:
    > > https://github.com/apache/incubator-mxnet/issues/11775
    > > https://github.com/apache/incubator-mxnet/issues/10753
    > > https://github.com/apache/incubator-mxnet/issues/11571
    > >
    > >
    >
    
    
    -- 
    Sandeep Krishnamurthy

Re: Data Type improvement in MXNet Scala

Posted by sandeep krishnamurthy <sa...@gmail.com>.

Hi Qing,
Thanks for picking this issue.
How would multiple inputs/labels be handled? By name or by position?

Background:
Recently I faced issue in Python where name of the input symbol name was
not used, but, position was used. Python2 reads list from 0 and Python3
reads list from last element. So based on the python version, I was having
swapped symbol and hence shape mis match errors.

Best
Sandeep

On Wed, Jul 18, 2018 at 8:09 PM Naveen Swamy <mn...@gmail.com> wrote:

> Thanks for looking into this Qing :)
>
> this would alleviate the problems we have in getting RNNs to work in Scala.
> Can we call name the new APIs `provideDataDesc` and `provideLabelDesc` ?
>
> Thanks, Naveen
>
> On Wed, Jul 18, 2018 at 3:43 PM, Qing Lan <la...@live.com> wrote:
>
> > Hi all,
> >
> > I would like to propose a change on the MXNet Scala data_type:
> >
> > Depreciate the `provideData` and `provideLabel` functions to provide
> > `DataDesc` type instead of `ListMap[String, Shape]` type.
> >
> > There were several issues regarding to this since v0.12 on Scala. Several
> > users reported that the `implicit def ListMap2Descs` helps to do some
> > downcasting job however missing information during this process (layout
> > property).
> > MXNet Python language has already change this part into `DataDesc`. In
> > that case, I plan to add two more methods that will take `DataDesc` and
> > depreciate the functions that mentioned above. The downcasting method
> > `ListMap2Descs` will also be depreciated and not to be used in current
> > MXNet Scala code base.
> >
> > Thanks,
> > Qing
> >
> > References:
> > Code position:
> > https://github.com/apache/incubator-mxnet/blob/master/
> > scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L312-L315
> > https://github.com/apache/incubator-mxnet/blob/master/
> > scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L358-L363
> >
> > Issue links:
> > https://github.com/apache/incubator-mxnet/issues/11775
> > https://github.com/apache/incubator-mxnet/issues/10753
> > https://github.com/apache/incubator-mxnet/issues/11571
> >
> >
>


-- 
Sandeep Krishnamurthy

Re: Data Type improvement in MXNet Scala

Posted by Naveen Swamy <mn...@gmail.com>.

Thanks for looking into this Qing :)

this would alleviate the problems we have in getting RNNs to work in Scala.
Can we call name the new APIs `provideDataDesc` and `provideLabelDesc` ?

Thanks, Naveen

On Wed, Jul 18, 2018 at 3:43 PM, Qing Lan <la...@live.com> wrote:

> Hi all,
>
> I would like to propose a change on the MXNet Scala data_type:
>
> Depreciate the `provideData` and `provideLabel` functions to provide
> `DataDesc` type instead of `ListMap[String, Shape]` type.
>
> There were several issues regarding to this since v0.12 on Scala. Several
> users reported that the `implicit def ListMap2Descs` helps to do some
> downcasting job however missing information during this process (layout
> property).
> MXNet Python language has already change this part into `DataDesc`. In
> that case, I plan to add two more methods that will take `DataDesc` and
> depreciate the functions that mentioned above. The downcasting method
> `ListMap2Descs` will also be depreciated and not to be used in current
> MXNet Scala code base.
>
> Thanks,
> Qing
>
> References:
> Code position:
> https://github.com/apache/incubator-mxnet/blob/master/
> scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L312-L315
> https://github.com/apache/incubator-mxnet/blob/master/
> scala-package/core/src/main/scala/org/apache/mxnet/IO.scala#L358-L363
>
> Issue links:
> https://github.com/apache/incubator-mxnet/issues/11775
> https://github.com/apache/incubator-mxnet/issues/10753
> https://github.com/apache/incubator-mxnet/issues/11571
>
>