You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2018/11/04 20:47:51 UTC
[GitHub] gigasquid closed pull request #12881: Improve the Clojure Package
README to Make it Easier to Get Started
gigasquid closed pull request #12881: Improve the Clojure Package README to Make it Easier to Get Started
URL: https://github.com/apache/incubator-mxnet/pull/12881
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/contrib/clojure-package/README.md b/contrib/clojure-package/README.md
index 8c71224a3a5..d5ee6ccbeb4 100644
--- a/contrib/clojure-package/README.md
+++ b/contrib/clojure-package/README.md
@@ -1,154 +1,204 @@
# Clojure MXNet
-A clojure package to the MXNet Deep Learning library
+A Clojure Package Built on the MXNet Deep Learning Library
## Introduction
-MXNet is a first class, modern deep learning library. It supports multiple languages on a first class basis and is incubating as an Apache project.
+MXNet is a flexible and efficient deep learning library. While its core is built in C++ for maximum performance, support for multiple programming languages in intermediate and high-level APIs is a first-class feature. MXNet is currently an incubating Apache project.
-The motivation for creating a Clojure package is to be able to open the deep learning library to the Clojure ecosystem and build bridges for future development and innovation for the community. It provides all the needed tools including low level and high level apis, dynamic graphs, and things like GAN and natural language support.
+The motivation for creating a Clojure package was to give Clojurians access to a world-class deep learning platform, thereby building bridges for future development and innovation in the community. The Clojure package provides all the essential tools, including low-level and high-level APIs, dynamic graphs, etc., and enables building advanced architectures like GANs or LSTM to tackle challenging applications such as image recognition or natural language processing.
-For high leverage, the Clojure package has been built on the existing Scala package using interop. This has allowed rapid development and close parity with the Scala functionality. This also leaves the door open to directly developing code against the jni-bindings with Clojure in the future in an incremental fashion, using the test suites as a refactoring guide.
+To maximize leverage, the Clojure package has been built on the existing Scala package using [Java Interop](https://clojure.org/reference/java_interop). This approach has allowed rapid initial development and close parity with the Scala package functionality. It also leaves the door open to incrementally developing Clojure code that directly interfaces MXNet core using [JNI](https://en.wikipedia.org/wiki/Java_Native_Interface).
-For a **video introduction**, see [Clojure MXNet with Carin Meier - Clojure Virtual Meetup](https://www.crowdcast.io/e/clojure-mxnet-with-carin) (setup instructions from 20:49)
+For a **video introduction**, see [Clojure MXNet with Carin Meier - Clojure Virtual Meetup](https://www.crowdcast.io/e/clojure-mxnet-with-carin) (setup instructions from 20:49).
## Current State and Plans
-The Clojure package is nearing the end of its first development milestone which is to achieve a close parity with the Scala package.
+The Clojure MXNet package is currently treated as *user-contributed code* within MXNet, as can be seen from its placement under `contrib` in the source tree. This means that it should first undergo a stabilization period and receive feedback from users before it can graduate to a fully integrated and supported part of MXNet.
-Help is needed testing and generally making the package better. A list of the pacakge status and contribution needs can be found here [Clojure Package Contribution Needs](https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs). Please get involved :)
+That said, because it closely tracks the Scala package, Clojure MXNet can be expected to have a similar level of maturity and stability regarding the low-level functionality. It is mostly in the hand-written Java interop part of the Clojure wrapper where bugs are more likely to be encountered. Such bugs tend to be fixed rather quickly once they are known and their origin is clear (see also [Getting Involved](#getting-involved)).
-Testing instructions can be found in the testing.md.
+For an overview of the development status and open problems, please refer to [Clojure Package Contribution Needs](https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs).
-## Getting Started
+## Getting Involved
-The following systems are supported:
+By far the best way to get involved with this project is to install the Clojure MXNet package, run the examples, play around, build new things with it, and get back to the development team with feedback! Your input can not only help to identify current issues, but also guide the future development of the Clojure package by pointing out must-have features that are currently missing, or by identifying usability or performace problems of high impact.
-- OSX cpu
-- Linux cpu
-- Linux gpu
+There are two main ways of reaching out to other users and the package maintainers:
-There are two ways of getting going. The first way is the easiest and that is to use the pre-built jars from Maven. The second way is to build from source. In both cases, you will need to load the prereqs and dependencies, (like opencv).
+- If you have a question or general feedback, or you encountered a problem but are not sure if it's a bug or a misunderstanding, then the *Apache Slack* (channels `#mxnet` and `#mxnet-scala`) is the best place to turn check out. To join, [ask for an invitation](https://mxnet.apache.org/community/contribute.html#slack) at `dev@mxnet.apache.org`.
+- If you found a bug, miss an important feature or want to give feedback directly relevant for development, please head over to the MXNet [GitHub issue page](https://github.com/apache/incubator-mxnet/issues) and create a new issue. If the issue is specific to the Clojure package, consider using a title starting with `[Clojure]` to make it easily discoverable among the many other, mostly generic issues.
+Of course, contributions to code or documentation are also more than welcome! Please check out the [Clojure Package Contribution Needs](https://cwiki.apache.org/confluence/display/MXNET/Clojure+Package+Contribution+Needs) to get an idea about where and how to contribute code.
+For a more comprehensive overview of different ways to contribute, see [Contributing to MXNet](https://mxnet.apache.org/community/contribute.html).
-### Prerequisites
+## Getting Started
+The Clojure MXNet framework consists of a core C library, a Scala API that talks to the core through [JNI (Java Native Interface)](https://en.wikipedia.org/wiki/Java_Native_Interface) bindings, and finally a Clojure wrapper around the Scala API.
-Follow the instructions from https://mxnet.incubator.apache.org/install/osx_setup.html or https://mxnet.incubator.apache.org/install/ubuntu_setup.html
-about _Prepare Environment for GPU Installation_
-and _Install MXNet dependencies_
+Since the core contains native (compiled) code and is bundled with the language bindings, your hardware and OS matter to the choices to be made during installation. The following combinations of operating system and compute device are supported:
+- Linux CPU
+- Linux GPU
+- OSX CPU
-#### Cloning the repo and running from source
+There are three ways of getting started:
-To use the prebuilt jars (easiest), you will need to replace the native version of the line in the project dependencies with your configuration.
+1. [Install prebuilt Clojure jars](#option-1-clojure-package-from-prebuilt-jar) with the native dependencies baked in. This the quickest way to get going.
+2. [Install the Clojure package from source, but use prebuilt jars for the native dependencies](#option-2-clojure-package-from-source-scala-package-from-jar). Choose this option if you want pre-release features of the Clojure package but don't want to build (compile) native dependencies yourself.
+3. [Build everything from source](#option-3-everything-from-source). This option is for developers or advanced users who want cutting-edge features in all parts of the dependency chain.
-`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-gpu "1.3.0"]`
-or
-`[org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.3.0"]`
-or
-`[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.3.0"]`
+**Note:** This guide assumes that you are familiar with the basics of creating Clojure projects and managing dependencies. See [here](https://github.com/technomancy/leiningen/blob/stable/doc/TUTORIAL.md) for the official Leiningen tutorial.
-If you are using the prebuilt jars they may have a slightly different dependencies then building from source:
+### Option 1: Clojure Package from Prebuilt Jar
-*For OSX you will need:*
+If you are new to MXNet and just want to try things out, this option is the best way to get started. You will install release versions of MXNet core, MXNet Scala and MXNet Clojure.
-`brew install opencv`
+For reference, the Clojure MXNet jars can be found on [maven.org](https://search.maven.org/search?q=clojure%20mxnet).
-*For Ubuntu Linux you will need:*
+#### Installing additional dependencies
-```
-sudo add-apt-repository ppa:timsc/opencv-3.4
+Depending on your operating system, you will need a couple of packages that are not distributed through Maven:
+
+- [OpenCV](https://opencv.org/) version 3.4
+- [OpenBLAS](https://www.openblas.net/)
+- [ATLAS](http://math-atlas.sourceforge.net/)
+- [cURL](https://curl.haxx.se/) library version 3
+
+##### Linux (Ubuntu)
+
+As of writing this, OpenCV 3.4 is not available in the default repositories. Therefore, a third-party repository is needed.
+
+```bash
+sudo add-apt-repository ppa:timsc/opencv
sudo apt-get update
-sudo apt install libopencv-imgcodecs3.4
+sudo apt install libopencv-imgcodecs3.4 libopenblas-base libatlas3-base libcurl3
```
-*For Arch Linux you will need:*
+Note: `libcurl3` may conflict with other packages on your system. [Here](https://github.com/apache/incubator-mxnet/issues/12822) is a possible workaround.
-_CPU_
+##### Linux (Arch)
-```
+```bash
yaourt -S openblas-lapack
yaourt -S libcurl-compat
export LD_PRELOAD=libcurl.so.3
```
-_GPU_
-```
+To enable GPU support, you will additionally need the CUDA toolkit:
+
+```bash
wget https://archive.archlinux.org/packages/c/cuda/cuda-9.0.176-4-x86_64.pkg.tar.xz
sudo pacman -U cuda-9.0.176-4-x86_64.pkg.tar.xz
```
-If you want to see the exact versions and flags that the jars were built with, look here:
-[Scala Release Process](https://cwiki.apache.org/confluence/display/MXNET/MXNet-Scala+Release+Process)
+##### OSX
+```bash
+brew install wget
+brew install opencv
+```
-Check your installation with `lein test`. If that works alright then, you can try some code!
+#### Installing the Clojure package
-```clojure
+- Create a new project with `lein new my-mxnet`
+- Edit your `project.clj` and add one of the following entries to `:dependencies`, based on your system and the compute device you want to use:
-(ns tutorial.ndarray
- (:require [org.apache.clojure-mxnet.ndarray :as ndarray]
- [org.apache.clojure-mxnet.context :as context]))
+ - `[org.apache.mxnet.contrib.clojure/clojure-mxnet-linux-cpu "1.3.0"]`
+ - `[org.apache.mxnet.contrib.clojure/clojure-mxnet-linux-gpu "1.3.0"]`
+ - `[org.apache.mxnet.contrib.clojure/clojure-mxnet-osx-cpu "1.3.0"]`
-;;Create NDArray
-(def a (ndarray/zeros [100 50])) ;;all zero arrray of dimension 100 x 50
-(def b (ndarray/ones [256 32 128 1])) ;; all one array of dimension
-(def c (ndarray/array [1 2 3 4 5 6] [2 3])) ;; array with contents of a shape 2 x 3
+After making this change and running `lein deps`, you should be able to run example code like this [NDArray Tutorial](https://github.com/apache/incubator-mxnet/blob/master/contrib/clojure-package/examples/tutorial/src/tutorial/ndarray.clj).
-;;; There are also ways to convert to a vec or get the shape as an object or vec
-(ndarray/->vec c) ;=> [1.0 2.0 3.0 4.0 5.0 6.0]
-```
+### Option 2: Clojure package from Source, Scala Package from Jar
+
+With this option, you will install a Git revision of the Clojure package source and a [Scala package jar from Maven](https://search.maven.org/search?q=g:org.apache.mxnet) with native dependencies baked in.
+
+- Install additional dependencies as described in [the corresponding section for Option 1](#installing-additional-dependencies),
+- Recursively clone the MXNet repository and checkout the desired revision. Here we assume the `1.3.0` tag and a clone into the `~/mxnet` directory:
+
+ ```bash
+ git clone --recursive https://github.com/apache/incubator-mxnet.git ~/mxnet
+ cd ~/mxnet
+ git tag --list # Find the tag that matches the Scala package version
+ git checkout tags/1.3.0 -b my_mxnet
+ git submodule update --init --recursive
+ cd contrib/clojure
+ ```
+
+- Edit `project.clj` to include the desired Scala jar from Maven:
-See the examples/tutorial section for more.
+ [org.apache.mxnet/mxnet-full_2.11-linux-x86_64-cpu "1.3.0”]
+- Run `lein test`. All the tests should run without error.
+- At this point you can run `lein install` to build and install the Clojure jar locally.
-The jars from maven with the needed MXNet native binaries in it. On startup, the native libraries are extracted from the jar and copied into a temporary location on your path. On termination, they are deleted.
+To run examples, you can now use `lein run` in any of the example directories, e.g., `examples/imclassification`. You can also specify the compute device, e.g., `lein run :cpu 2` (for 2 CPUs) or `lein run :gpu` (for 1 GPU).
+**Note:** Instead of a release tag, you can also use a development version of the Clojure package, e.g., Git `master`, together with the prebuilt Scala jar. In that case, however, breakage can happen at any point, for instance when the Scala development version adds, changes or removes an interface and the Clojure development version moves along. If you really need the most recent version, you should consider [installation option 3](#option-3-everything-from-source).
-### Build from MXNET Source
+### Option 3: Everything from Source
-First, ensure you have JDK 8 on your system. Later versions may produce cryptic build errors mentioning `scala.reflect.internal.MissingRequirementError`.
+With this option, you will compile the core MXNet C++ package and jars for both Scala and Clojure language bindings from source. If you intend to make changes to the code in any of the parts, or if you simply want the latest and greatest features, this choice is for you.
-Checkout the latest SHA from the main package:
+The first step is to recursively clone the MXNet repository and checkout the desired revision. Here we assume a clone into the `~/mxnet` directory:
-`git clone --recursive https://github.com/apache/incubator-mxnet.git ~/mxnet`
-`cd ~/mxnet`
+ ```bash
+ git clone --recursive https://github.com/apache/incubator-mxnet.git ~/mxnet
+ cd ~/mxnet
+ git checkout tags/1.3.0 -b my_mxnet # this is optional
+ git submodule update --init --recursive
+ ```
-If you need to checkout a particular release you can do it with:
+If you have previous builds and other unwanted files lying around in the working directory and would like to clean up, [here](https://gist.github.com/nicktoumpelis/11214362) is a useful script for that task. However, be aware that this recipe will remove any untracked files and reset uncommitted changes in the working directory.
-`git checkout tags/1.3.0 -b release-1.3.0`
+#### Building the core library
-`git submodule update --init --recursive`
+Detailed instructions for building MXNet core from source can be found [in the MXNet installation documentation](https://mxnet.incubator.apache.org/install/index.html). The relevant sections are:
-Sometimes it useful to use this script to clean hard
-https://gist.github.com/nicktoumpelis/11214362
+- For Ubuntu Linux: [CUDA Dependencies](https://mxnet.incubator.apache.org/install/ubuntu_setup.html#cuda-dependencies) and [Building MXNet from Source](https://mxnet.incubator.apache.org/install/ubuntu_setup.html#build-mxnet-from-source)
+- For Mac OSX: [Build the Shared Library](https://mxnet.incubator.apache.org/install/osx_setup.html#build-the-shared-library)
+In particular, ignore all of the language-interface-specific sections.
-Go here to do the base package installation https://mxnet.incubator.apache.org/install/index.html
+The outcome of this step will be a shared library `lib/libmxnet.so` that is used in the next step.
- Run `make scalapkg` then `make scalainstall`
+#### Building the Scala jar
-then replace the correct jar for your architecture in the project.clj, example `[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.3.0-SNAPSHOT"]`
+- Ensure you have JDK 8 on your system. Later versions may produce cryptic build errors mentioning `scala.reflect.internal.MissingRequirementError`.
+- Build and install the Scala package in your local Maven directory using the following commands:
-#### Test your installation
+ ```bash
+ make scalapkg
+ make scalainstall
+ ```
-To test your installation, you should run `lein test`. This will run the test suite (CPU) for the clojure package.
+#### Building the Clojure jar
+
+- Enter the `contrib/clojure` directory and edit the `project.clj` file. Add the Scala jar that was just created and installed, e.g., `[org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.3.0-SNAPSHOT"]`, to the `:dependencies`.
+- Run `lein test`. All the tests should run without an error.
+- Run `lein install` to build and install the Clojure jar locally.
+To run examples, you can now use `lein run` in any of the example directories, e.g., `examples/imclassification`. You can also specify the compute device, e.g., `lein run :cpu 2` (for 2 CPUs) or `lein run :gpu` (for 1 GPU).
-#### Generation of NDArray and Symbol apis
+## Docker Files
-The bulk of the ndarray and symbol apis are generated via java reflection into the Scala classes. The files are generated as a compile time step (AOT) in the `dev.generator` namespace.
+There are Dockerfiles available as well.
-You may also run this manually with the repl functions:
+- [Community Provided by Magnet](https://hub.docker.com/u/magnetcoop/)
+- [MXNet CI](https://github.com/apache/incubator-mxnet/blob/master/ci/docker/Dockerfile.build.ubuntu_cpu) and the install scripts
+ - [Ubuntu core](https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_core.sh)
+ - [Ubuntu Scala](https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_scala.sh)
+ - [Ubuntu Clojure](https://github.com/apache/incubator-mxnet/blob/master/ci/docker/install/ubuntu_clojure.sh)
-`(generate-ndarray-file)`
-and
-`(generate-symbol-file)`
+## Need Help?
+If you are having trouble getting started or have a question, feel free to reach out at:
-These will generate the files under `src/org.apache.clojure-mxnet/gen/` that are loaded by the `src/org.apache.clojure-mxnet/ndarray.clj` and `src/org.apache.clojure-mxnet/symbol.clj` files.
+- Clojurian Slack #mxnet channel. To join, go to [http://clojurians.net/](http://clojurians.net/).
+- Apache Slack #mxnet and #mxnet-scala channel. To join this slack send an email to dev@mxnet.apache.org.
+- Create an Issue on [https://github.com/apache/incubator-mxnet/issues](https://github.com/apache/incubator-mxnet/issues).
## Examples
@@ -160,11 +210,11 @@ There are quite a few examples in the examples directory. To use.
There are README is every directory outlining instructions.
A good place to get started is the module example.
-Do `lein run` for the cpu version or `lein run :gpu` for gpu.
+Do `lein run` for the CPU version or `lein run :gpu` for GPU.
## Generating documentation
-To generate api docs, run `lein codox`. The html docs will be generated in the target/docs directory.
+To generate API docs, run `lein codox`. The html docs will be generated in the target/docs directory.
## Code Coverage
@@ -182,12 +232,11 @@ before the submit a new pull request so we can keep the style consistent through
## FAQ
-
**Why build on the Scala package?**
The motivation section addresses this, but the main reason is high leverage is using the great work that the Scala package has already done.
-**How can I tell if the gpu is being used?**
+**How can I tell if the GPU is being used?**
CUDA is finding a best algorithm... As long as a Context.gpu() passed in the code as a context, GPU should be used.
@@ -197,30 +246,17 @@ This command can be very handy too
timestamp, name, utilization.gpu [%], utilization.memory [%], memory.total [MiB], memory.free [MiB], memory.used [MiB]`
**Supported APIs**
-There are 3 high level apis supported in MXNet: (Model/FeedForward), Module, and Gluon. The Module api is supported in the Clojure package because of the existing support for it in the Scala package. The Module api is very similar to the Gluon api and examples of the usage can be found in the examples directory. The Model/FeedForward Api is deprected.
+There are 3 high level APIs supported in MXNet: (Model/FeedForward), Module, and Gluon. The Module API is supported in the Clojure package because of the existing support for it in the Scala package. The Module API is very similar to the Gluon API and examples of the usage can be found in the examples directory. The Model/FeedForward API is deprected.
-Gluon support will come later and may or may not be built on the Scala gluon api (when it lands there)
+Gluon support will come later and may or may not be built on the Scala gluon API (when it lands there)
## Architecture & Design
See the Confluence page: https://cwiki.apache.org/confluence/display/MXNET/MXNet+Clojure
## Building and Deploying Jars
-The process to build and deploy the jars currently is a manual process using the `lein` build tool and `Clojars`, the Clojure dependency hosting platform.
-
-There is one jar for every system supported.
-
-- Comment out the line in the `project.clj` for the system that you are targeting, (example OSX cpu you would uncomment out ` [org.apache.mxnet/mxnet-full_2.11-osx-x86_64-cpu "1.2.0"]` but leave the linux deps commented)
-- Change the `defproject org.apache.mxnet.contrib.clojure/clojure-mxnet "0.1.1-SNAPSHOT"` in the project to reference the correct version number and jar description. For example changing the line to be `org.apache.mxnet.contrib.clojure/mxnet-osx-cpu "0.1.2"` would create a jar with the group id of `org.apache.mxnet.contrib.clojure` and the artifact name of `mxnet-osx-cpu` and the version of `0.1.2`
-- Run `lein clean`
-- Run `lein jar` to create the jar
-- Check that the jar looks alright in the `/target` directory.
-
-To deploy the jar to Clojars, you do `lein deploy clojars` and it will prompt you for your username and password.
-
-_Note: Integration with deployment to Nexus can be enabled too for the future [https://help.sonatype.com/repomanager2/maven-and-other-build-tools/leiningen](https://help.sonatype.com/repomanager2/maven-and-other-build-tools/leiningen)_
-You would repeat this process for all the build system types.
+The release process for deploying the Clojure jars is on the [Apache MXNet developer wiki](https://cwiki.apache.org/confluence/display/MXNET/Clojure+Release+Process).
## Special Thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services