You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by YiZhi Liu <li...@apache.org> on 2018/10/29 17:39:27 UTC
Apache MXNet Scala nightly-build now available on Apache Nexus
Hi,
I am happy to announce that the availability of the experimental
nightly-build Scala package on Maven in Nexus. The nightly-builds,
currently published as 1.3.1-SNAPSHOT version, include the latest changes
from the master branch from apache/incubator-mxnet GitHub repo and refresh
every day. Furthermore, we completely overhauled the MXNet library shipped
inside and adopted the binary build logic used for Python pip, so now Scala
users can also enjoy the portable MXNet with fully loaded features.
Currently, the supported artifacts are:
- mxnet-full_2.11-linux-x86_64-cpu (Linux-CPU)
- mxnet-full_2.11-linux-x86_64-gpu (Linux-GPU, CUDA 9.0)
- mxnet-full_2.11-osx-x86_64-cpu (OSX-CPU)
FAQ:
- What's new in the nightly Scala packages compared to the previous Scala
Maven artifacts?
Most notably, better usability, more supported features, and better
platform compatibility. The new packages now support Ubuntu 14.04 and
Amazon Linux and support the same broad set of features as the existing pip
packages. Besides, users are no longer required to install all the
dependencies to use the nightly Scala packages as the included MXNet
libraries are fully loaded. For the list of supported platforms and
features, see [1].
- What's different in the build approach compared to what's in the previous
Scala Maven packages?
The previous Scala Maven packages (1.2, 1.2.1, 1.3) were built with the
same build commands that regular users would use when building from source
[2]. This naive approach limits the compatibility to only similar
environments as where the packages were built (which was Ubuntu 16.04 for
Linux, and OSX 10.13 for OSX). As a result, users cannot use it on more
CentOS or Amazon Linux and may suffer the problem of mismatching of
dependency versions due to the difference in their environment.
On the other hand, Python users have long been enjoying portable and fully
loaded MXNet packages from pip. The MXNet libraries in pip packages are
built in more controlled environments with compatibility in mind.
Dependencies for extended features are statically linked and masked so that
no additional steps are needed.
Now, in the new Scala nightly packages, the same approach as pip is used
for building the MXNet library so that the Scala users can benefit from the
same great features.
- How do I get the new nightly packages?
You can find the packages on Apache (Snapshots) repository [3] including
pom files and jars, which you can download and add to your projects.
Alternatively, you can configure your pom.xml to use the Maven repository.
Remember to add repository config:
<repositories>
<repository>
<id>Apache Snapshot</id>
<url>https://repository.apache.org/content/groups/snapshots</url>
</repository>
</repositories>
- What's the support level of the Scala nightly packages?
This package is currently experimental, so use at your own risk. While the
first nightly packages are already manually verified on several platforms,
and the automated publish pipeline runs unit tests and integration tests on
OSX and Ubuntu 14.04 before publishing, the testing is currently limited.
- How can I help?
We call on all MXNet Scala community to test out the new nightly packages.
- I found a problem in using the package. What should I do?
Please report this in the Github issue in [1].
- What's next?
There are several more things to be done:
1. We are adding more comprehensive support for GPU versions for different
CUDA versions.
2. We are adding more tests on more platforms. We are currently examining
the MXNet's Jenkins CI solution for supporting more platforms.
3. As we recognize that the build logic benefits the community, we plan to
open source the build scripts in the mxnet main repo. See progress at [5].
Qing Lan, Zach Kimberg, Sheng Zha, and I worked together to make this
happen. Thanks to Sheng Zha for sharing the pip build logic, and for
building the automated publishing pipelines. Special thanks to Qing and
Zach for sharing their expertise in Scala with Sheng and guiding and
supporting him locally for this milestone.
[1] https://github.com/apache/incubator-mxnet/issues/8671
[2]
https://github.com/apache/incubator-mxnet/blob/master/scala-package/dev/compile-mxnet-backend.sh#L59-L105
[3]
https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~1.3.1-SNAPSHOT~~
[4] https://maven.apache.org/repository-management.html
[5] https://github.com/apache/incubator-mxnet/projects/13
Re: Apache MXNet Scala nightly-build now available on Apache Nexus
Posted by Ziheng Jiang <zi...@apache.org>.
Nice work! It is quite helpful for deployment of MXNet Scala!
On Mon, Oct 29, 2018 at 10:39 AM YiZhi Liu <li...@apache.org> wrote:
> Hi,
>
>
>
> I am happy to announce that the availability of the experimental
> nightly-build Scala package on Maven in Nexus. The nightly-builds,
> currently published as 1.3.1-SNAPSHOT version, include the latest changes
> from the master branch from apache/incubator-mxnet GitHub repo and refresh
> every day. Furthermore, we completely overhauled the MXNet library shipped
> inside and adopted the binary build logic used for Python pip, so now Scala
> users can also enjoy the portable MXNet with fully loaded features.
> Currently, the supported artifacts are:
>
> - mxnet-full_2.11-linux-x86_64-cpu (Linux-CPU)
>
> - mxnet-full_2.11-linux-x86_64-gpu (Linux-GPU, CUDA 9.0)
>
> - mxnet-full_2.11-osx-x86_64-cpu (OSX-CPU)
>
>
>
> FAQ:
>
> - What's new in the nightly Scala packages compared to the previous Scala
> Maven artifacts?
>
> Most notably, better usability, more supported features, and better
> platform compatibility. The new packages now support Ubuntu 14.04 and
> Amazon Linux and support the same broad set of features as the existing pip
> packages. Besides, users are no longer required to install all the
> dependencies to use the nightly Scala packages as the included MXNet
> libraries are fully loaded. For the list of supported platforms and
> features, see [1].
>
>
>
> - What's different in the build approach compared to what's in the previous
> Scala Maven packages?
>
> The previous Scala Maven packages (1.2, 1.2.1, 1.3) were built with the
> same build commands that regular users would use when building from source
> [2]. This naive approach limits the compatibility to only similar
> environments as where the packages were built (which was Ubuntu 16.04 for
> Linux, and OSX 10.13 for OSX). As a result, users cannot use it on more
> CentOS or Amazon Linux and may suffer the problem of mismatching of
> dependency versions due to the difference in their environment.
>
> On the other hand, Python users have long been enjoying portable and fully
> loaded MXNet packages from pip. The MXNet libraries in pip packages are
> built in more controlled environments with compatibility in mind.
> Dependencies for extended features are statically linked and masked so that
> no additional steps are needed.
>
> Now, in the new Scala nightly packages, the same approach as pip is used
> for building the MXNet library so that the Scala users can benefit from the
> same great features.
>
>
>
> - How do I get the new nightly packages?
>
> You can find the packages on Apache (Snapshots) repository [3] including
> pom files and jars, which you can download and add to your projects.
> Alternatively, you can configure your pom.xml to use the Maven repository.
> Remember to add repository config:
>
> <repositories>
>
> <repository>
>
> <id>Apache Snapshot</id>
>
> <url>https://repository.apache.org/content/groups/snapshots</url>
>
> </repository>
>
> </repositories>
>
>
>
> - What's the support level of the Scala nightly packages?
>
> This package is currently experimental, so use at your own risk. While the
> first nightly packages are already manually verified on several platforms,
> and the automated publish pipeline runs unit tests and integration tests on
> OSX and Ubuntu 14.04 before publishing, the testing is currently limited.
>
>
>
> - How can I help?
>
> We call on all MXNet Scala community to test out the new nightly packages.
>
>
>
> - I found a problem in using the package. What should I do?
>
> Please report this in the Github issue in [1].
>
>
>
> - What's next?
>
> There are several more things to be done:
>
> 1. We are adding more comprehensive support for GPU versions for different
> CUDA versions.
>
> 2. We are adding more tests on more platforms. We are currently examining
> the MXNet's Jenkins CI solution for supporting more platforms.
>
> 3. As we recognize that the build logic benefits the community, we plan to
> open source the build scripts in the mxnet main repo. See progress at [5].
>
>
>
> Qing Lan, Zach Kimberg, Sheng Zha, and I worked together to make this
> happen. Thanks to Sheng Zha for sharing the pip build logic, and for
> building the automated publishing pipelines. Special thanks to Qing and
> Zach for sharing their expertise in Scala with Sheng and guiding and
> supporting him locally for this milestone.
>
>
>
> [1] https://github.com/apache/incubator-mxnet/issues/8671
>
> [2]
>
> https://github.com/apache/incubator-mxnet/blob/master/scala-package/dev/compile-mxnet-backend.sh#L59-L105
>
> [3]
>
> https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~1.3.1-SNAPSHOT~~
>
> [4] https://maven.apache.org/repository-management.html
>
> [5] https://github.com/apache/incubator-mxnet/projects/13
>
--
Ziheng Jiang
Fudan University | Computer Science
Re: Apache MXNet Scala nightly-build now available on Apache Nexus
Posted by YiZhi Liu <li...@apache.org>.
Thanks for attention. Here's an example demonstrates how to configure
pom.xml to use the nightly build in your own project:
https://github.com/yzhliu/mxnet-scala-example/blob/master/pom.xml#L152-L163
On Mon, Oct 29, 2018 at 2:53 PM Carin Meier <ca...@gmail.com> wrote:
>
> Fantastic! This is going to be great for the Clojure MXNet users as well.
>
> On Mon, Oct 29, 2018 at 1:39 PM YiZhi Liu <li...@apache.org> wrote:
>
> > Hi,
> >
> >
> >
> > I am happy to announce that the availability of the experimental
> > nightly-build Scala package on Maven in Nexus. The nightly-builds,
> > currently published as 1.3.1-SNAPSHOT version, include the latest changes
> > from the master branch from apache/incubator-mxnet GitHub repo and refresh
> > every day. Furthermore, we completely overhauled the MXNet library shipped
> > inside and adopted the binary build logic used for Python pip, so now Scala
> > users can also enjoy the portable MXNet with fully loaded features.
> > Currently, the supported artifacts are:
> >
> > - mxnet-full_2.11-linux-x86_64-cpu (Linux-CPU)
> >
> > - mxnet-full_2.11-linux-x86_64-gpu (Linux-GPU, CUDA 9.0)
> >
> > - mxnet-full_2.11-osx-x86_64-cpu (OSX-CPU)
> >
> >
> >
> > FAQ:
> >
> > - What's new in the nightly Scala packages compared to the previous Scala
> > Maven artifacts?
> >
> > Most notably, better usability, more supported features, and better
> > platform compatibility. The new packages now support Ubuntu 14.04 and
> > Amazon Linux and support the same broad set of features as the existing pip
> > packages. Besides, users are no longer required to install all the
> > dependencies to use the nightly Scala packages as the included MXNet
> > libraries are fully loaded. For the list of supported platforms and
> > features, see [1].
> >
> >
> >
> > - What's different in the build approach compared to what's in the previous
> > Scala Maven packages?
> >
> > The previous Scala Maven packages (1.2, 1.2.1, 1.3) were built with the
> > same build commands that regular users would use when building from source
> > [2]. This naive approach limits the compatibility to only similar
> > environments as where the packages were built (which was Ubuntu 16.04 for
> > Linux, and OSX 10.13 for OSX). As a result, users cannot use it on more
> > CentOS or Amazon Linux and may suffer the problem of mismatching of
> > dependency versions due to the difference in their environment.
> >
> > On the other hand, Python users have long been enjoying portable and fully
> > loaded MXNet packages from pip. The MXNet libraries in pip packages are
> > built in more controlled environments with compatibility in mind.
> > Dependencies for extended features are statically linked and masked so that
> > no additional steps are needed.
> >
> > Now, in the new Scala nightly packages, the same approach as pip is used
> > for building the MXNet library so that the Scala users can benefit from the
> > same great features.
> >
> >
> >
> > - How do I get the new nightly packages?
> >
> > You can find the packages on Apache (Snapshots) repository [3] including
> > pom files and jars, which you can download and add to your projects.
> > Alternatively, you can configure your pom.xml to use the Maven repository.
> > Remember to add repository config:
> >
> > <repositories>
> >
> > <repository>
> >
> > <id>Apache Snapshot</id>
> >
> > <url>https://repository.apache.org/content/groups/snapshots</url>
> >
> > </repository>
> >
> > </repositories>
> >
> >
> >
> > - What's the support level of the Scala nightly packages?
> >
> > This package is currently experimental, so use at your own risk. While the
> > first nightly packages are already manually verified on several platforms,
> > and the automated publish pipeline runs unit tests and integration tests on
> > OSX and Ubuntu 14.04 before publishing, the testing is currently limited.
> >
> >
> >
> > - How can I help?
> >
> > We call on all MXNet Scala community to test out the new nightly packages.
> >
> >
> >
> > - I found a problem in using the package. What should I do?
> >
> > Please report this in the Github issue in [1].
> >
> >
> >
> > - What's next?
> >
> > There are several more things to be done:
> >
> > 1. We are adding more comprehensive support for GPU versions for different
> > CUDA versions.
> >
> > 2. We are adding more tests on more platforms. We are currently examining
> > the MXNet's Jenkins CI solution for supporting more platforms.
> >
> > 3. As we recognize that the build logic benefits the community, we plan to
> > open source the build scripts in the mxnet main repo. See progress at [5].
> >
> >
> >
> > Qing Lan, Zach Kimberg, Sheng Zha, and I worked together to make this
> > happen. Thanks to Sheng Zha for sharing the pip build logic, and for
> > building the automated publishing pipelines. Special thanks to Qing and
> > Zach for sharing their expertise in Scala with Sheng and guiding and
> > supporting him locally for this milestone.
> >
> >
> >
> > [1] https://github.com/apache/incubator-mxnet/issues/8671
> >
> > [2]
> >
> > https://github.com/apache/incubator-mxnet/blob/master/scala-package/dev/compile-mxnet-backend.sh#L59-L105
> >
> > [3]
> >
> > https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~1.3.1-SNAPSHOT~~
> >
> > [4] https://maven.apache.org/repository-management.html
> >
> > [5] https://github.com/apache/incubator-mxnet/projects/13
> >
--
Yizhi Liu
DMLC member
Amazon Web Services
Vancouver, Canada
Re: Apache MXNet Scala nightly-build now available on Apache Nexus
Posted by Carin Meier <ca...@gmail.com>.
Fantastic! This is going to be great for the Clojure MXNet users as well.
On Mon, Oct 29, 2018 at 1:39 PM YiZhi Liu <li...@apache.org> wrote:
> Hi,
>
>
>
> I am happy to announce that the availability of the experimental
> nightly-build Scala package on Maven in Nexus. The nightly-builds,
> currently published as 1.3.1-SNAPSHOT version, include the latest changes
> from the master branch from apache/incubator-mxnet GitHub repo and refresh
> every day. Furthermore, we completely overhauled the MXNet library shipped
> inside and adopted the binary build logic used for Python pip, so now Scala
> users can also enjoy the portable MXNet with fully loaded features.
> Currently, the supported artifacts are:
>
> - mxnet-full_2.11-linux-x86_64-cpu (Linux-CPU)
>
> - mxnet-full_2.11-linux-x86_64-gpu (Linux-GPU, CUDA 9.0)
>
> - mxnet-full_2.11-osx-x86_64-cpu (OSX-CPU)
>
>
>
> FAQ:
>
> - What's new in the nightly Scala packages compared to the previous Scala
> Maven artifacts?
>
> Most notably, better usability, more supported features, and better
> platform compatibility. The new packages now support Ubuntu 14.04 and
> Amazon Linux and support the same broad set of features as the existing pip
> packages. Besides, users are no longer required to install all the
> dependencies to use the nightly Scala packages as the included MXNet
> libraries are fully loaded. For the list of supported platforms and
> features, see [1].
>
>
>
> - What's different in the build approach compared to what's in the previous
> Scala Maven packages?
>
> The previous Scala Maven packages (1.2, 1.2.1, 1.3) were built with the
> same build commands that regular users would use when building from source
> [2]. This naive approach limits the compatibility to only similar
> environments as where the packages were built (which was Ubuntu 16.04 for
> Linux, and OSX 10.13 for OSX). As a result, users cannot use it on more
> CentOS or Amazon Linux and may suffer the problem of mismatching of
> dependency versions due to the difference in their environment.
>
> On the other hand, Python users have long been enjoying portable and fully
> loaded MXNet packages from pip. The MXNet libraries in pip packages are
> built in more controlled environments with compatibility in mind.
> Dependencies for extended features are statically linked and masked so that
> no additional steps are needed.
>
> Now, in the new Scala nightly packages, the same approach as pip is used
> for building the MXNet library so that the Scala users can benefit from the
> same great features.
>
>
>
> - How do I get the new nightly packages?
>
> You can find the packages on Apache (Snapshots) repository [3] including
> pom files and jars, which you can download and add to your projects.
> Alternatively, you can configure your pom.xml to use the Maven repository.
> Remember to add repository config:
>
> <repositories>
>
> <repository>
>
> <id>Apache Snapshot</id>
>
> <url>https://repository.apache.org/content/groups/snapshots</url>
>
> </repository>
>
> </repositories>
>
>
>
> - What's the support level of the Scala nightly packages?
>
> This package is currently experimental, so use at your own risk. While the
> first nightly packages are already manually verified on several platforms,
> and the automated publish pipeline runs unit tests and integration tests on
> OSX and Ubuntu 14.04 before publishing, the testing is currently limited.
>
>
>
> - How can I help?
>
> We call on all MXNet Scala community to test out the new nightly packages.
>
>
>
> - I found a problem in using the package. What should I do?
>
> Please report this in the Github issue in [1].
>
>
>
> - What's next?
>
> There are several more things to be done:
>
> 1. We are adding more comprehensive support for GPU versions for different
> CUDA versions.
>
> 2. We are adding more tests on more platforms. We are currently examining
> the MXNet's Jenkins CI solution for supporting more platforms.
>
> 3. As we recognize that the build logic benefits the community, we plan to
> open source the build scripts in the mxnet main repo. See progress at [5].
>
>
>
> Qing Lan, Zach Kimberg, Sheng Zha, and I worked together to make this
> happen. Thanks to Sheng Zha for sharing the pip build logic, and for
> building the automated publishing pipelines. Special thanks to Qing and
> Zach for sharing their expertise in Scala with Sheng and guiding and
> supporting him locally for this milestone.
>
>
>
> [1] https://github.com/apache/incubator-mxnet/issues/8671
>
> [2]
>
> https://github.com/apache/incubator-mxnet/blob/master/scala-package/dev/compile-mxnet-backend.sh#L59-L105
>
> [3]
>
> https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~1.3.1-SNAPSHOT~~
>
> [4] https://maven.apache.org/repository-management.html
>
> [5] https://github.com/apache/incubator-mxnet/projects/13
>
Re: Apache MXNet Scala nightly-build now available on Apache Nexus
Posted by Tianqi Chen <tq...@cs.washington.edu>.
This is a great step for the community
Tianqi
On Mon, Oct 29, 2018 at 10:39 AM YiZhi Liu <li...@apache.org> wrote:
> Hi,
>
>
>
> I am happy to announce that the availability of the experimental
> nightly-build Scala package on Maven in Nexus. The nightly-builds,
> currently published as 1.3.1-SNAPSHOT version, include the latest changes
> from the master branch from apache/incubator-mxnet GitHub repo and refresh
> every day. Furthermore, we completely overhauled the MXNet library shipped
> inside and adopted the binary build logic used for Python pip, so now Scala
> users can also enjoy the portable MXNet with fully loaded features.
> Currently, the supported artifacts are:
>
> - mxnet-full_2.11-linux-x86_64-cpu (Linux-CPU)
>
> - mxnet-full_2.11-linux-x86_64-gpu (Linux-GPU, CUDA 9.0)
>
> - mxnet-full_2.11-osx-x86_64-cpu (OSX-CPU)
>
>
>
> FAQ:
>
> - What's new in the nightly Scala packages compared to the previous Scala
> Maven artifacts?
>
> Most notably, better usability, more supported features, and better
> platform compatibility. The new packages now support Ubuntu 14.04 and
> Amazon Linux and support the same broad set of features as the existing pip
> packages. Besides, users are no longer required to install all the
> dependencies to use the nightly Scala packages as the included MXNet
> libraries are fully loaded. For the list of supported platforms and
> features, see [1].
>
>
>
> - What's different in the build approach compared to what's in the previous
> Scala Maven packages?
>
> The previous Scala Maven packages (1.2, 1.2.1, 1.3) were built with the
> same build commands that regular users would use when building from source
> [2]. This naive approach limits the compatibility to only similar
> environments as where the packages were built (which was Ubuntu 16.04 for
> Linux, and OSX 10.13 for OSX). As a result, users cannot use it on more
> CentOS or Amazon Linux and may suffer the problem of mismatching of
> dependency versions due to the difference in their environment.
>
> On the other hand, Python users have long been enjoying portable and fully
> loaded MXNet packages from pip. The MXNet libraries in pip packages are
> built in more controlled environments with compatibility in mind.
> Dependencies for extended features are statically linked and masked so that
> no additional steps are needed.
>
> Now, in the new Scala nightly packages, the same approach as pip is used
> for building the MXNet library so that the Scala users can benefit from the
> same great features.
>
>
>
> - How do I get the new nightly packages?
>
> You can find the packages on Apache (Snapshots) repository [3] including
> pom files and jars, which you can download and add to your projects.
> Alternatively, you can configure your pom.xml to use the Maven repository.
> Remember to add repository config:
>
> <repositories>
>
> <repository>
>
> <id>Apache Snapshot</id>
>
> <url>https://repository.apache.org/content/groups/snapshots</url>
>
> </repository>
>
> </repositories>
>
>
>
> - What's the support level of the Scala nightly packages?
>
> This package is currently experimental, so use at your own risk. While the
> first nightly packages are already manually verified on several platforms,
> and the automated publish pipeline runs unit tests and integration tests on
> OSX and Ubuntu 14.04 before publishing, the testing is currently limited.
>
>
>
> - How can I help?
>
> We call on all MXNet Scala community to test out the new nightly packages.
>
>
>
> - I found a problem in using the package. What should I do?
>
> Please report this in the Github issue in [1].
>
>
>
> - What's next?
>
> There are several more things to be done:
>
> 1. We are adding more comprehensive support for GPU versions for different
> CUDA versions.
>
> 2. We are adding more tests on more platforms. We are currently examining
> the MXNet's Jenkins CI solution for supporting more platforms.
>
> 3. As we recognize that the build logic benefits the community, we plan to
> open source the build scripts in the mxnet main repo. See progress at [5].
>
>
>
> Qing Lan, Zach Kimberg, Sheng Zha, and I worked together to make this
> happen. Thanks to Sheng Zha for sharing the pip build logic, and for
> building the automated publishing pipelines. Special thanks to Qing and
> Zach for sharing their expertise in Scala with Sheng and guiding and
> supporting him locally for this milestone.
>
>
>
> [1] https://github.com/apache/incubator-mxnet/issues/8671
>
> [2]
>
> https://github.com/apache/incubator-mxnet/blob/master/scala-package/dev/compile-mxnet-backend.sh#L59-L105
>
> [3]
>
> https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~1.3.1-SNAPSHOT~~
>
> [4] https://maven.apache.org/repository-management.html
>
> [5] https://github.com/apache/incubator-mxnet/projects/13
>