You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by Frank Liu <fr...@gmail.com> on 2019/07/03 00:50:17 UTC
[DISCUSS] Make MXNet deploy it's own distribution
Currently, MXNet were built along with different language bindings such as
Scala.
The libmxnet.so files are bundled within scala jar package.
It would be nice to distribute libmxnet.so library independently in maven,
and scala package can choose which native library to use.
Here is the design document on cwiki:
https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
Thanks,
Frank
Re: [DISCUSS] Make MXNet deploy it's own distribution
Posted by Carin Meier <ca...@gmail.com>.
From the Clojure package perspective, since it is compatible with maven,
this approach will work fine.
It would also make it easier for developers to build on top of MXNet and
share their libraries.
- Carin
On Wed, Jul 3, 2019 at 3:45 AM Per da Silva <pe...@gmail.com> wrote:
> Hi,
>
> We've started working on something along these lines as part of the CD
> pipeline framework. The idea is to compile and test the libmxnet.so (both
> statically and dynamically linked) for the different variants (cpu, gpu,
> mkl, etc.) then have the different mxnet frontends (python, Julia, scala,
> etc) just wrap around the library.
>
> I've been on long term sick leave and haven't been able to move forward
> with this, although I have an open PR that kicks off this work:
> https://github.com/apache/incubator-mxnet/pull/15051 - I welcome everyone
> to take a look. It's the first of a series of PRs to automate the
> distribution of the python (pip and docker) packages. Instead of using
> maven, we have opted to use S3. But this decision can be revisited.
>
> We also want to distribute what we termed "runtime" docker images. Docker
> images containing the dynamically linked mxnet library and all of the
> runtime dependencies (examples: https://hub.docker.com/r/mxnet/runtime).
> This could facilitate the packaging and distribution of docker images for
> the different frontends.
>
> Cheers,
>
> Per
>
> On Wed., 3 Jul. 2019, 8:47 am Qing Lan, <la...@live.com> wrote:
>
> > In that case, the answer is yes. The Scala package will be published in
> > one version with a variaty of backend package choices. User can easily
> > attach and detach different MXNet versions. However, the Scala package
> > cannot run without a backend.
> >
> > Another key advantage of this design will be a broader support on
> > different implementations such as Java Cpp. User will be able to
> implement
> > their customized MXNet frontend to use the native library.
> >
> > Thanks,
> > Qing
> >
> > ________________________________
> > From: Sheng Zha <sz...@gmail.com>
> > Sent: Tuesday, July 2, 2019 22:14
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: [DISCUSS] Make MXNet deploy it's own distribution
> >
> > Does it mean that the scala binding of mxnet will be an independent
> > package that doesn’t directly depend on the native package, and user
> > projects need to declare dependency on both the scala binding and one of
> > the native packages?
> >
> > -sz
> >
> > > On Jul 2, 2019, at 5:50 PM, Frank Liu <fr...@gmail.com> wrote:
> > >
> > > Currently, MXNet were built along with different language bindings such
> > as
> > > Scala.
> > >
> > > The libmxnet.so files are bundled within scala jar package.
> > >
> > > It would be nice to distribute libmxnet.so library independently in
> > maven,
> > > and scala package can choose which native library to use.
> > >
> > > Here is the design document on cwiki:
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
> > >
> > > Thanks,
> > >
> > > Frank
> >
>
Re: [DISCUSS] Make MXNet deploy it's own distribution
Posted by Pedro Larroy <pe...@gmail.com>.
Nice! +1 To this approach, seems well thought. Thanks for including
Android and linux-arm. Does Android and linux-arm use a different
classifier?
On Wed, Jul 3, 2019 at 6:46 AM Chris Olivier <cj...@apache.org> wrote:
>
> Will this be another repo under Apache repo? Is tensorflow java package in
> a separate repo?
>
> On Wed, Jul 3, 2019 at 12:46 AM Per da Silva <pe...@gmail.com> wrote:
>
> > Hi,
> >
> > We've started working on something along these lines as part of the CD
> > pipeline framework. The idea is to compile and test the libmxnet.so (both
> > statically and dynamically linked) for the different variants (cpu, gpu,
> > mkl, etc.) then have the different mxnet frontends (python, Julia, scala,
> > etc) just wrap around the library.
> >
> > I've been on long term sick leave and haven't been able to move forward
> > with this, although I have an open PR that kicks off this work:
> > https://github.com/apache/incubator-mxnet/pull/15051 - I welcome everyone
> > to take a look. It's the first of a series of PRs to automate the
> > distribution of the python (pip and docker) packages. Instead of using
> > maven, we have opted to use S3. But this decision can be revisited.
> >
> > We also want to distribute what we termed "runtime" docker images. Docker
> > images containing the dynamically linked mxnet library and all of the
> > runtime dependencies (examples: https://hub.docker.com/r/mxnet/runtime).
> > This could facilitate the packaging and distribution of docker images for
> > the different frontends.
> >
> > Cheers,
> >
> > Per
> >
> > On Wed., 3 Jul. 2019, 8:47 am Qing Lan, <la...@live.com> wrote:
> >
> > > In that case, the answer is yes. The Scala package will be published in
> > > one version with a variaty of backend package choices. User can easily
> > > attach and detach different MXNet versions. However, the Scala package
> > > cannot run without a backend.
> > >
> > > Another key advantage of this design will be a broader support on
> > > different implementations such as Java Cpp. User will be able to
> > implement
> > > their customized MXNet frontend to use the native library.
> > >
> > > Thanks,
> > > Qing
> > >
> > > ________________________________
> > > From: Sheng Zha <sz...@gmail.com>
> > > Sent: Tuesday, July 2, 2019 22:14
> > > To: dev@mxnet.incubator.apache.org
> > > Subject: Re: [DISCUSS] Make MXNet deploy it's own distribution
> > >
> > > Does it mean that the scala binding of mxnet will be an independent
> > > package that doesn’t directly depend on the native package, and user
> > > projects need to declare dependency on both the scala binding and one of
> > > the native packages?
> > >
> > > -sz
> > >
> > > > On Jul 2, 2019, at 5:50 PM, Frank Liu <fr...@gmail.com> wrote:
> > > >
> > > > Currently, MXNet were built along with different language bindings such
> > > as
> > > > Scala.
> > > >
> > > > The libmxnet.so files are bundled within scala jar package.
> > > >
> > > > It would be nice to distribute libmxnet.so library independently in
> > > maven,
> > > > and scala package can choose which native library to use.
> > > >
> > > > Here is the design document on cwiki:
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
> > > >
> > > > Thanks,
> > > >
> > > > Frank
> > >
> >
Re: [DISCUSS] Make MXNet deploy it's own distribution
Posted by Frank Liu <fr...@gmail.com>.
No, This just a CD pipeline that publish a jar files contains libmxnet.so
to maven central.
We are publishing Scala package to maven central already. The issue with
current Scala maven package is not flexible to developer who want's switch
different version of libmxnet.so file.
With this proposal, developer can easily choose different libmxnet.so file
by auto detect the os type.
On Wed, Jul 3, 2019 at 6:46 AM Chris Olivier <cj...@apache.org> wrote:
> Will this be another repo under Apache repo? Is tensorflow java package in
> a separate repo?
>
> On Wed, Jul 3, 2019 at 12:46 AM Per da Silva <pe...@gmail.com> wrote:
>
> > Hi,
> >
> > We've started working on something along these lines as part of the CD
> > pipeline framework. The idea is to compile and test the libmxnet.so
> (both
> > statically and dynamically linked) for the different variants (cpu, gpu,
> > mkl, etc.) then have the different mxnet frontends (python, Julia, scala,
> > etc) just wrap around the library.
> >
> > I've been on long term sick leave and haven't been able to move forward
> > with this, although I have an open PR that kicks off this work:
> > https://github.com/apache/incubator-mxnet/pull/15051 - I welcome
> everyone
> > to take a look. It's the first of a series of PRs to automate the
> > distribution of the python (pip and docker) packages. Instead of using
> > maven, we have opted to use S3. But this decision can be revisited.
> >
> > We also want to distribute what we termed "runtime" docker images. Docker
> > images containing the dynamically linked mxnet library and all of the
> > runtime dependencies (examples: https://hub.docker.com/r/mxnet/runtime).
> > This could facilitate the packaging and distribution of docker images for
> > the different frontends.
> >
> > Cheers,
> >
> > Per
> >
> > On Wed., 3 Jul. 2019, 8:47 am Qing Lan, <la...@live.com> wrote:
> >
> > > In that case, the answer is yes. The Scala package will be published in
> > > one version with a variaty of backend package choices. User can easily
> > > attach and detach different MXNet versions. However, the Scala package
> > > cannot run without a backend.
> > >
> > > Another key advantage of this design will be a broader support on
> > > different implementations such as Java Cpp. User will be able to
> > implement
> > > their customized MXNet frontend to use the native library.
> > >
> > > Thanks,
> > > Qing
> > >
> > > ________________________________
> > > From: Sheng Zha <sz...@gmail.com>
> > > Sent: Tuesday, July 2, 2019 22:14
> > > To: dev@mxnet.incubator.apache.org
> > > Subject: Re: [DISCUSS] Make MXNet deploy it's own distribution
> > >
> > > Does it mean that the scala binding of mxnet will be an independent
> > > package that doesn’t directly depend on the native package, and user
> > > projects need to declare dependency on both the scala binding and one
> of
> > > the native packages?
> > >
> > > -sz
> > >
> > > > On Jul 2, 2019, at 5:50 PM, Frank Liu <fr...@gmail.com>
> wrote:
> > > >
> > > > Currently, MXNet were built along with different language bindings
> such
> > > as
> > > > Scala.
> > > >
> > > > The libmxnet.so files are bundled within scala jar package.
> > > >
> > > > It would be nice to distribute libmxnet.so library independently in
> > > maven,
> > > > and scala package can choose which native library to use.
> > > >
> > > > Here is the design document on cwiki:
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
> > > >
> > > > Thanks,
> > > >
> > > > Frank
> > >
> >
>
Re: [DISCUSS] Make MXNet deploy it's own distribution
Posted by Chris Olivier <cj...@apache.org>.
Will this be another repo under Apache repo? Is tensorflow java package in
a separate repo?
On Wed, Jul 3, 2019 at 12:46 AM Per da Silva <pe...@gmail.com> wrote:
> Hi,
>
> We've started working on something along these lines as part of the CD
> pipeline framework. The idea is to compile and test the libmxnet.so (both
> statically and dynamically linked) for the different variants (cpu, gpu,
> mkl, etc.) then have the different mxnet frontends (python, Julia, scala,
> etc) just wrap around the library.
>
> I've been on long term sick leave and haven't been able to move forward
> with this, although I have an open PR that kicks off this work:
> https://github.com/apache/incubator-mxnet/pull/15051 - I welcome everyone
> to take a look. It's the first of a series of PRs to automate the
> distribution of the python (pip and docker) packages. Instead of using
> maven, we have opted to use S3. But this decision can be revisited.
>
> We also want to distribute what we termed "runtime" docker images. Docker
> images containing the dynamically linked mxnet library and all of the
> runtime dependencies (examples: https://hub.docker.com/r/mxnet/runtime).
> This could facilitate the packaging and distribution of docker images for
> the different frontends.
>
> Cheers,
>
> Per
>
> On Wed., 3 Jul. 2019, 8:47 am Qing Lan, <la...@live.com> wrote:
>
> > In that case, the answer is yes. The Scala package will be published in
> > one version with a variaty of backend package choices. User can easily
> > attach and detach different MXNet versions. However, the Scala package
> > cannot run without a backend.
> >
> > Another key advantage of this design will be a broader support on
> > different implementations such as Java Cpp. User will be able to
> implement
> > their customized MXNet frontend to use the native library.
> >
> > Thanks,
> > Qing
> >
> > ________________________________
> > From: Sheng Zha <sz...@gmail.com>
> > Sent: Tuesday, July 2, 2019 22:14
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: [DISCUSS] Make MXNet deploy it's own distribution
> >
> > Does it mean that the scala binding of mxnet will be an independent
> > package that doesn’t directly depend on the native package, and user
> > projects need to declare dependency on both the scala binding and one of
> > the native packages?
> >
> > -sz
> >
> > > On Jul 2, 2019, at 5:50 PM, Frank Liu <fr...@gmail.com> wrote:
> > >
> > > Currently, MXNet were built along with different language bindings such
> > as
> > > Scala.
> > >
> > > The libmxnet.so files are bundled within scala jar package.
> > >
> > > It would be nice to distribute libmxnet.so library independently in
> > maven,
> > > and scala package can choose which native library to use.
> > >
> > > Here is the design document on cwiki:
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
> > >
> > > Thanks,
> > >
> > > Frank
> >
>
Re: [DISCUSS] Make MXNet deploy it's own distribution
Posted by Frank Liu <fr...@gmail.com>.
maven packages are crucial to JVM based language. This proposal can
definitely benefit from your PR. Without your PR, maven publish can not be
automated.
It would be nice to automate maven publish as party of your CD pipeline.
On Wed, Jul 3, 2019 at 12:45 AM Per da Silva <pe...@gmail.com> wrote:
> Hi,
>
> We've started working on something along these lines as part of the CD
> pipeline framework. The idea is to compile and test the libmxnet.so (both
> statically and dynamically linked) for the different variants (cpu, gpu,
> mkl, etc.) then have the different mxnet frontends (python, Julia, scala,
> etc) just wrap around the library.
>
> I've been on long term sick leave and haven't been able to move forward
> with this, although I have an open PR that kicks off this work:
> https://github.com/apache/incubator-mxnet/pull/15051 - I welcome everyone
> to take a look. It's the first of a series of PRs to automate the
> distribution of the python (pip and docker) packages. Instead of using
> maven, we have opted to use S3. But this decision can be revisited.
>
> We also want to distribute what we termed "runtime" docker images. Docker
> images containing the dynamically linked mxnet library and all of the
> runtime dependencies (examples: https://hub.docker.com/r/mxnet/runtime).
> This could facilitate the packaging and distribution of docker images for
> the different frontends.
>
> Cheers,
>
> Per
>
> On Wed., 3 Jul. 2019, 8:47 am Qing Lan, <la...@live.com> wrote:
>
> > In that case, the answer is yes. The Scala package will be published in
> > one version with a variaty of backend package choices. User can easily
> > attach and detach different MXNet versions. However, the Scala package
> > cannot run without a backend.
> >
> > Another key advantage of this design will be a broader support on
> > different implementations such as Java Cpp. User will be able to
> implement
> > their customized MXNet frontend to use the native library.
> >
> > Thanks,
> > Qing
> >
> > ________________________________
> > From: Sheng Zha <sz...@gmail.com>
> > Sent: Tuesday, July 2, 2019 22:14
> > To: dev@mxnet.incubator.apache.org
> > Subject: Re: [DISCUSS] Make MXNet deploy it's own distribution
> >
> > Does it mean that the scala binding of mxnet will be an independent
> > package that doesn’t directly depend on the native package, and user
> > projects need to declare dependency on both the scala binding and one of
> > the native packages?
> >
> > -sz
> >
> > > On Jul 2, 2019, at 5:50 PM, Frank Liu <fr...@gmail.com> wrote:
> > >
> > > Currently, MXNet were built along with different language bindings such
> > as
> > > Scala.
> > >
> > > The libmxnet.so files are bundled within scala jar package.
> > >
> > > It would be nice to distribute libmxnet.so library independently in
> > maven,
> > > and scala package can choose which native library to use.
> > >
> > > Here is the design document on cwiki:
> > >
> >
> https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
> > >
> > > Thanks,
> > >
> > > Frank
> >
>
Re: [DISCUSS] Make MXNet deploy it's own distribution
Posted by Per da Silva <pe...@gmail.com>.
Hi,
We've started working on something along these lines as part of the CD
pipeline framework. The idea is to compile and test the libmxnet.so (both
statically and dynamically linked) for the different variants (cpu, gpu,
mkl, etc.) then have the different mxnet frontends (python, Julia, scala,
etc) just wrap around the library.
I've been on long term sick leave and haven't been able to move forward
with this, although I have an open PR that kicks off this work:
https://github.com/apache/incubator-mxnet/pull/15051 - I welcome everyone
to take a look. It's the first of a series of PRs to automate the
distribution of the python (pip and docker) packages. Instead of using
maven, we have opted to use S3. But this decision can be revisited.
We also want to distribute what we termed "runtime" docker images. Docker
images containing the dynamically linked mxnet library and all of the
runtime dependencies (examples: https://hub.docker.com/r/mxnet/runtime).
This could facilitate the packaging and distribution of docker images for
the different frontends.
Cheers,
Per
On Wed., 3 Jul. 2019, 8:47 am Qing Lan, <la...@live.com> wrote:
> In that case, the answer is yes. The Scala package will be published in
> one version with a variaty of backend package choices. User can easily
> attach and detach different MXNet versions. However, the Scala package
> cannot run without a backend.
>
> Another key advantage of this design will be a broader support on
> different implementations such as Java Cpp. User will be able to implement
> their customized MXNet frontend to use the native library.
>
> Thanks,
> Qing
>
> ________________________________
> From: Sheng Zha <sz...@gmail.com>
> Sent: Tuesday, July 2, 2019 22:14
> To: dev@mxnet.incubator.apache.org
> Subject: Re: [DISCUSS] Make MXNet deploy it's own distribution
>
> Does it mean that the scala binding of mxnet will be an independent
> package that doesn’t directly depend on the native package, and user
> projects need to declare dependency on both the scala binding and one of
> the native packages?
>
> -sz
>
> > On Jul 2, 2019, at 5:50 PM, Frank Liu <fr...@gmail.com> wrote:
> >
> > Currently, MXNet were built along with different language bindings such
> as
> > Scala.
> >
> > The libmxnet.so files are bundled within scala jar package.
> >
> > It would be nice to distribute libmxnet.so library independently in
> maven,
> > and scala package can choose which native library to use.
> >
> > Here is the design document on cwiki:
> >
> https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
> >
> > Thanks,
> >
> > Frank
>
Re: [DISCUSS] Make MXNet deploy it's own distribution
Posted by Qing Lan <la...@live.com>.
In that case, the answer is yes. The Scala package will be published in one version with a variaty of backend package choices. User can easily attach and detach different MXNet versions. However, the Scala package cannot run without a backend.
Another key advantage of this design will be a broader support on different implementations such as Java Cpp. User will be able to implement their customized MXNet frontend to use the native library.
Thanks,
Qing
________________________________
From: Sheng Zha <sz...@gmail.com>
Sent: Tuesday, July 2, 2019 22:14
To: dev@mxnet.incubator.apache.org
Subject: Re: [DISCUSS] Make MXNet deploy it's own distribution
Does it mean that the scala binding of mxnet will be an independent package that doesn’t directly depend on the native package, and user projects need to declare dependency on both the scala binding and one of the native packages?
-sz
> On Jul 2, 2019, at 5:50 PM, Frank Liu <fr...@gmail.com> wrote:
>
> Currently, MXNet were built along with different language bindings such as
> Scala.
>
> The libmxnet.so files are bundled within scala jar package.
>
> It would be nice to distribute libmxnet.so library independently in maven,
> and scala package can choose which native library to use.
>
> Here is the design document on cwiki:
> https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
>
> Thanks,
>
> Frank
Re: [DISCUSS] Make MXNet deploy it's own distribution
Posted by Sheng Zha <sz...@gmail.com>.
Does it mean that the scala binding of mxnet will be an independent package that doesn’t directly depend on the native package, and user projects need to declare dependency on both the scala binding and one of the native packages?
-sz
> On Jul 2, 2019, at 5:50 PM, Frank Liu <fr...@gmail.com> wrote:
>
> Currently, MXNet were built along with different language bindings such as
> Scala.
>
> The libmxnet.so files are bundled within scala jar package.
>
> It would be nice to distribute libmxnet.so library independently in maven,
> and scala package can choose which native library to use.
>
> Here is the design document on cwiki:
> https://cwiki.apache.org/confluence/display/MXNET/Make+MXNet+deploy+it%27s+own+distribution
>
> Thanks,
>
> Frank