You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mxnet.apache.org by Pedro Larroy <pe...@gmail.com> on 2017/11/02 14:57:04 UTC
update build instructions
Hi
I would like to update the MXNet build instructions.
In particular I was thinking that it would be a good idea to update
the instructions to use CMake + Ninja. And add more information about
the different build flavours.
https://mxnet.incubator.apache.org/install/index.html
Thoughts?
Re: update build instructions
Posted by Chris Olivier <cj...@gmail.com>.
I understand that people may be using it, but what purpose does it serve
that CMake does not?
Making something that triggers cmake transparently from a 'make' command in
the root dir (just run it from GNUMakefile, for example) and puts
everything in the same place is a trivial task. What does using the
Makefil offer that cmake does not?
On Thu, Nov 2, 2017 at 8:57 AM, Jun Wu <wu...@gmail.com> wrote:
> Many of developers are using Makefile. Getting rid of it doesn't sound user
> friendly.
>
> On Thu, Nov 2, 2017 at 8:53 AM, Chris Olivier <cj...@gmail.com>
> wrote:
>
> > I don't know why we don't get rid of the Makefile altogether and use
> cmake.
> > It's a pain to manage both of them independently. Does anyone know why
> we
> > still use the Makefile?
> >
> > On Thu, Nov 2, 2017 at 8:45 AM, Pedro Larroy <
> pedro.larroy.lists@gmail.com
> > >
> > wrote:
> >
> > > Hi
> > >
> > > For me it's more about correctness and reproducibility than build
> > > times, nonetheless, seems that the ninja build is significantly faster
> > > than the Make build:
> > >
> > > Make:
> > >
> > > real 4m32.779s
> > > user 43m33.236s
> > > sys 0m52.940s
> > >
> > > CMake + Ninja:
> > >
> > > real 3m30.794s
> > > user 36m2.564s
> > > sys 0m56.368s
> > >
> > > Compiled on an g3.4xlarge with ebs
> > >
> > > Provisioned IOPS SSD
> > >
> > > io (115000 iops)
> > >
> > >
> > > Pedro.
> > >
> > > On Thu, Nov 2, 2017 at 4:07 PM, Bhavin Thaker <bh...@gmail.com>
> > > wrote:
> > > > Hi Pedro,
> > > >
> > > > Using Ninja to improve build times is a good suggestion. Can you
> share
> > > the
> > > > build times you have observed with and without using Ninja? I presume
> > you
> > > > have enabled compile-time options for GPU builds and Distributed
> MXNet
> > > for
> > > > the builds you have experimented with.
> > > >
> > > > See also:
> > > > https://ninja-build.org/manual.html
> > > >
> > > > Thanks,
> > > > Bhavin Thaker.
> > > >
> > > > On Thu, Nov 2, 2017 at 7:57 AM, Pedro Larroy <
> > > pedro.larroy.lists@gmail.com>
> > > > wrote:
> > > >
> > > >> Hi
> > > >>
> > > >> I would like to update the MXNet build instructions.
> > > >>
> > > >> In particular I was thinking that it would be a good idea to update
> > > >> the instructions to use CMake + Ninja. And add more information
> about
> > > >> the different build flavours.
> > > >>
> > > >>
> > > >> https://mxnet.incubator.apache.org/install/index.html
> > > >>
> > > >>
> > > >> Thoughts?
> > > >>
> > >
> >
>
Re: update build instructions
Posted by Jun Wu <wu...@gmail.com>.
Many of developers are using Makefile. Getting rid of it doesn't sound user
friendly.
On Thu, Nov 2, 2017 at 8:53 AM, Chris Olivier <cj...@gmail.com> wrote:
> I don't know why we don't get rid of the Makefile altogether and use cmake.
> It's a pain to manage both of them independently. Does anyone know why we
> still use the Makefile?
>
> On Thu, Nov 2, 2017 at 8:45 AM, Pedro Larroy <pedro.larroy.lists@gmail.com
> >
> wrote:
>
> > Hi
> >
> > For me it's more about correctness and reproducibility than build
> > times, nonetheless, seems that the ninja build is significantly faster
> > than the Make build:
> >
> > Make:
> >
> > real 4m32.779s
> > user 43m33.236s
> > sys 0m52.940s
> >
> > CMake + Ninja:
> >
> > real 3m30.794s
> > user 36m2.564s
> > sys 0m56.368s
> >
> > Compiled on an g3.4xlarge with ebs
> >
> > Provisioned IOPS SSD
> >
> > io (115000 iops)
> >
> >
> > Pedro.
> >
> > On Thu, Nov 2, 2017 at 4:07 PM, Bhavin Thaker <bh...@gmail.com>
> > wrote:
> > > Hi Pedro,
> > >
> > > Using Ninja to improve build times is a good suggestion. Can you share
> > the
> > > build times you have observed with and without using Ninja? I presume
> you
> > > have enabled compile-time options for GPU builds and Distributed MXNet
> > for
> > > the builds you have experimented with.
> > >
> > > See also:
> > > https://ninja-build.org/manual.html
> > >
> > > Thanks,
> > > Bhavin Thaker.
> > >
> > > On Thu, Nov 2, 2017 at 7:57 AM, Pedro Larroy <
> > pedro.larroy.lists@gmail.com>
> > > wrote:
> > >
> > >> Hi
> > >>
> > >> I would like to update the MXNet build instructions.
> > >>
> > >> In particular I was thinking that it would be a good idea to update
> > >> the instructions to use CMake + Ninja. And add more information about
> > >> the different build flavours.
> > >>
> > >>
> > >> https://mxnet.incubator.apache.org/install/index.html
> > >>
> > >>
> > >> Thoughts?
> > >>
> >
>
Re: update build instructions
Posted by Chris Olivier <cj...@gmail.com>.
I don't know why we don't get rid of the Makefile altogether and use cmake.
It's a pain to manage both of them independently. Does anyone know why we
still use the Makefile?
On Thu, Nov 2, 2017 at 8:45 AM, Pedro Larroy <pe...@gmail.com>
wrote:
> Hi
>
> For me it's more about correctness and reproducibility than build
> times, nonetheless, seems that the ninja build is significantly faster
> than the Make build:
>
> Make:
>
> real 4m32.779s
> user 43m33.236s
> sys 0m52.940s
>
> CMake + Ninja:
>
> real 3m30.794s
> user 36m2.564s
> sys 0m56.368s
>
> Compiled on an g3.4xlarge with ebs
>
> Provisioned IOPS SSD
>
> io (115000 iops)
>
>
> Pedro.
>
> On Thu, Nov 2, 2017 at 4:07 PM, Bhavin Thaker <bh...@gmail.com>
> wrote:
> > Hi Pedro,
> >
> > Using Ninja to improve build times is a good suggestion. Can you share
> the
> > build times you have observed with and without using Ninja? I presume you
> > have enabled compile-time options for GPU builds and Distributed MXNet
> for
> > the builds you have experimented with.
> >
> > See also:
> > https://ninja-build.org/manual.html
> >
> > Thanks,
> > Bhavin Thaker.
> >
> > On Thu, Nov 2, 2017 at 7:57 AM, Pedro Larroy <
> pedro.larroy.lists@gmail.com>
> > wrote:
> >
> >> Hi
> >>
> >> I would like to update the MXNet build instructions.
> >>
> >> In particular I was thinking that it would be a good idea to update
> >> the instructions to use CMake + Ninja. And add more information about
> >> the different build flavours.
> >>
> >>
> >> https://mxnet.incubator.apache.org/install/index.html
> >>
> >>
> >> Thoughts?
> >>
>
Re: update build instructions
Posted by Pedro Larroy <pe...@gmail.com>.
Hi
I updated the build instructions using CMake, python3 and other minor
inacuracies in this PR:
https://github.com/apache/incubator-mxnet/pull/8578
Please have a look and comment.
I also added a file in the root "DEVELOPMENT.md" which synthesizes
how to build mxnet and run the unit tests to get started making
changes.
Your feedback is welcome. And if you can please test the instructions
to check they work properly and nothing was missed.
Pedro.
On Thu, Nov 2, 2017 at 5:07 PM, Pedro Larroy
<pe...@gmail.com> wrote:
> Right.
>
> I tried now with the flavor that you requested and I have problems
> generating the build file:
>
> Seems that I need the variable pslite_LINKER_LIBS_DEBUG which is not
> set. Any idea on how to compile with this flavour? (dist KVSTORE)
>
>
>
>
> ubuntu@ip-172-31-35-161:~/mxnet/build$ cmake -DUSE_CUDA=ON
> -DUSE_DIST_KVSTORE=ON -GNinja ..
> -- Found MKL (include: /usr/local/include, lib: /usr/local/lib/libmklml_gnu.so
> -- Found OpenBLAS libraries: /usr/local/lib/libopenblas.so
> -- Found OpenBLAS include: /usr/local/include
> -- CUDA detected: 8.0
> -- Found cuDNN (include: /usr/local/cuda/include, library:
> /usr/local/cuda/lib64/libcudnn.so)
> -- Added CUDA NVCC flags for: sm_52
> -- Could NOT find Gperftools (missing: GPERFTOOLS_LIBRARIES
> GPERFTOOLS_INCLUDE_DIR)
> -- Could NOT find Jemalloc (missing: JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR)
> -- OpenCV_LIBS=opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs
> -- OpenCV found (/usr/local/share/OpenCV)
> -- Could NOT find Jemalloc (missing: JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR)
> -- Found cuDNN (include: /usr/local/cuda/include, library:
> /usr/local/cuda/lib64/libcudnn.so)
> You have called ADD_LIBRARY for library mxnet without any source
> files. This typically indicates a problem with your CMakeLists.txt
> file
> -- Found PROTOBUF Compiler: /usr/local/bin/protoc
> CMake Error at CMakeLists.txt:446 (target_link_libraries):
> The "debug" argument must be followed by a library.
>
>
> -- Configuring incomplete, errors occurred!
> See also "/home/ubuntu/mxnet/build/CMakeFiles/CMakeOutput.log".
> See also "/home/ubuntu/mxnet/build/CMakeFiles/CMakeError.log".
>
>
> Regards.
>
> Pedro.
>
> On Thu, Nov 2, 2017 at 4:52 PM, Bhavin Thaker <bh...@gmail.com> wrote:
>> I agree about your point on correctness -- do you know of any known
>> correctness issues with Ninja?
>>
>> These build times seem to be NOT with GPU builds and distributed kvstore
>> enabled -- could you please confirm? nvcc builds take a significant time.
>>
>> Bhavin Thaker.
>>
>> On Thu, Nov 2, 2017 at 8:45 AM, Pedro Larroy <pe...@gmail.com>
>> wrote:
>>
>>> Hi
>>>
>>> For me it's more about correctness and reproducibility than build
>>> times, nonetheless, seems that the ninja build is significantly faster
>>> than the Make build:
>>>
>>> Make:
>>>
>>> real 4m32.779s
>>> user 43m33.236s
>>> sys 0m52.940s
>>>
>>> CMake + Ninja:
>>>
>>> real 3m30.794s
>>> user 36m2.564s
>>> sys 0m56.368s
>>>
>>> Compiled on an g3.4xlarge with ebs
>>>
>>> Provisioned IOPS SSD
>>>
>>> io (115000 iops)
>>>
>>>
>>> Pedro.
>>>
>>> On Thu, Nov 2, 2017 at 4:07 PM, Bhavin Thaker <bh...@gmail.com>
>>> wrote:
>>> > Hi Pedro,
>>> >
>>> > Using Ninja to improve build times is a good suggestion. Can you share
>>> the
>>> > build times you have observed with and without using Ninja? I presume you
>>> > have enabled compile-time options for GPU builds and Distributed MXNet
>>> for
>>> > the builds you have experimented with.
>>> >
>>> > See also:
>>> > https://ninja-build.org/manual.html
>>> >
>>> > Thanks,
>>> > Bhavin Thaker.
>>> >
>>> > On Thu, Nov 2, 2017 at 7:57 AM, Pedro Larroy <
>>> pedro.larroy.lists@gmail.com>
>>> > wrote:
>>> >
>>> >> Hi
>>> >>
>>> >> I would like to update the MXNet build instructions.
>>> >>
>>> >> In particular I was thinking that it would be a good idea to update
>>> >> the instructions to use CMake + Ninja. And add more information about
>>> >> the different build flavours.
>>> >>
>>> >>
>>> >> https://mxnet.incubator.apache.org/install/index.html
>>> >>
>>> >>
>>> >> Thoughts?
>>> >>
>>>
Re: update build instructions
Posted by Pedro Larroy <pe...@gmail.com>.
Right.
I tried now with the flavor that you requested and I have problems
generating the build file:
Seems that I need the variable pslite_LINKER_LIBS_DEBUG which is not
set. Any idea on how to compile with this flavour? (dist KVSTORE)
ubuntu@ip-172-31-35-161:~/mxnet/build$ cmake -DUSE_CUDA=ON
-DUSE_DIST_KVSTORE=ON -GNinja ..
-- Found MKL (include: /usr/local/include, lib: /usr/local/lib/libmklml_gnu.so
-- Found OpenBLAS libraries: /usr/local/lib/libopenblas.so
-- Found OpenBLAS include: /usr/local/include
-- CUDA detected: 8.0
-- Found cuDNN (include: /usr/local/cuda/include, library:
/usr/local/cuda/lib64/libcudnn.so)
-- Added CUDA NVCC flags for: sm_52
-- Could NOT find Gperftools (missing: GPERFTOOLS_LIBRARIES
GPERFTOOLS_INCLUDE_DIR)
-- Could NOT find Jemalloc (missing: JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR)
-- OpenCV_LIBS=opencv_core;opencv_highgui;opencv_imgproc;opencv_imgcodecs
-- OpenCV found (/usr/local/share/OpenCV)
-- Could NOT find Jemalloc (missing: JEMALLOC_LIBRARY JEMALLOC_INCLUDE_DIR)
-- Found cuDNN (include: /usr/local/cuda/include, library:
/usr/local/cuda/lib64/libcudnn.so)
You have called ADD_LIBRARY for library mxnet without any source
files. This typically indicates a problem with your CMakeLists.txt
file
-- Found PROTOBUF Compiler: /usr/local/bin/protoc
CMake Error at CMakeLists.txt:446 (target_link_libraries):
The "debug" argument must be followed by a library.
-- Configuring incomplete, errors occurred!
See also "/home/ubuntu/mxnet/build/CMakeFiles/CMakeOutput.log".
See also "/home/ubuntu/mxnet/build/CMakeFiles/CMakeError.log".
Regards.
Pedro.
On Thu, Nov 2, 2017 at 4:52 PM, Bhavin Thaker <bh...@gmail.com> wrote:
> I agree about your point on correctness -- do you know of any known
> correctness issues with Ninja?
>
> These build times seem to be NOT with GPU builds and distributed kvstore
> enabled -- could you please confirm? nvcc builds take a significant time.
>
> Bhavin Thaker.
>
> On Thu, Nov 2, 2017 at 8:45 AM, Pedro Larroy <pe...@gmail.com>
> wrote:
>
>> Hi
>>
>> For me it's more about correctness and reproducibility than build
>> times, nonetheless, seems that the ninja build is significantly faster
>> than the Make build:
>>
>> Make:
>>
>> real 4m32.779s
>> user 43m33.236s
>> sys 0m52.940s
>>
>> CMake + Ninja:
>>
>> real 3m30.794s
>> user 36m2.564s
>> sys 0m56.368s
>>
>> Compiled on an g3.4xlarge with ebs
>>
>> Provisioned IOPS SSD
>>
>> io (115000 iops)
>>
>>
>> Pedro.
>>
>> On Thu, Nov 2, 2017 at 4:07 PM, Bhavin Thaker <bh...@gmail.com>
>> wrote:
>> > Hi Pedro,
>> >
>> > Using Ninja to improve build times is a good suggestion. Can you share
>> the
>> > build times you have observed with and without using Ninja? I presume you
>> > have enabled compile-time options for GPU builds and Distributed MXNet
>> for
>> > the builds you have experimented with.
>> >
>> > See also:
>> > https://ninja-build.org/manual.html
>> >
>> > Thanks,
>> > Bhavin Thaker.
>> >
>> > On Thu, Nov 2, 2017 at 7:57 AM, Pedro Larroy <
>> pedro.larroy.lists@gmail.com>
>> > wrote:
>> >
>> >> Hi
>> >>
>> >> I would like to update the MXNet build instructions.
>> >>
>> >> In particular I was thinking that it would be a good idea to update
>> >> the instructions to use CMake + Ninja. And add more information about
>> >> the different build flavours.
>> >>
>> >>
>> >> https://mxnet.incubator.apache.org/install/index.html
>> >>
>> >>
>> >> Thoughts?
>> >>
>>
Re: update build instructions
Posted by Bhavin Thaker <bh...@gmail.com>.
I agree about your point on correctness -- do you know of any known
correctness issues with Ninja?
These build times seem to be NOT with GPU builds and distributed kvstore
enabled -- could you please confirm? nvcc builds take a significant time.
Bhavin Thaker.
On Thu, Nov 2, 2017 at 8:45 AM, Pedro Larroy <pe...@gmail.com>
wrote:
> Hi
>
> For me it's more about correctness and reproducibility than build
> times, nonetheless, seems that the ninja build is significantly faster
> than the Make build:
>
> Make:
>
> real 4m32.779s
> user 43m33.236s
> sys 0m52.940s
>
> CMake + Ninja:
>
> real 3m30.794s
> user 36m2.564s
> sys 0m56.368s
>
> Compiled on an g3.4xlarge with ebs
>
> Provisioned IOPS SSD
>
> io (115000 iops)
>
>
> Pedro.
>
> On Thu, Nov 2, 2017 at 4:07 PM, Bhavin Thaker <bh...@gmail.com>
> wrote:
> > Hi Pedro,
> >
> > Using Ninja to improve build times is a good suggestion. Can you share
> the
> > build times you have observed with and without using Ninja? I presume you
> > have enabled compile-time options for GPU builds and Distributed MXNet
> for
> > the builds you have experimented with.
> >
> > See also:
> > https://ninja-build.org/manual.html
> >
> > Thanks,
> > Bhavin Thaker.
> >
> > On Thu, Nov 2, 2017 at 7:57 AM, Pedro Larroy <
> pedro.larroy.lists@gmail.com>
> > wrote:
> >
> >> Hi
> >>
> >> I would like to update the MXNet build instructions.
> >>
> >> In particular I was thinking that it would be a good idea to update
> >> the instructions to use CMake + Ninja. And add more information about
> >> the different build flavours.
> >>
> >>
> >> https://mxnet.incubator.apache.org/install/index.html
> >>
> >>
> >> Thoughts?
> >>
>
Re: update build instructions
Posted by Pedro Larroy <pe...@gmail.com>.
Hi
For me it's more about correctness and reproducibility than build
times, nonetheless, seems that the ninja build is significantly faster
than the Make build:
Make:
real 4m32.779s
user 43m33.236s
sys 0m52.940s
CMake + Ninja:
real 3m30.794s
user 36m2.564s
sys 0m56.368s
Compiled on an g3.4xlarge with ebs
Provisioned IOPS SSD
io (115000 iops)
Pedro.
On Thu, Nov 2, 2017 at 4:07 PM, Bhavin Thaker <bh...@gmail.com> wrote:
> Hi Pedro,
>
> Using Ninja to improve build times is a good suggestion. Can you share the
> build times you have observed with and without using Ninja? I presume you
> have enabled compile-time options for GPU builds and Distributed MXNet for
> the builds you have experimented with.
>
> See also:
> https://ninja-build.org/manual.html
>
> Thanks,
> Bhavin Thaker.
>
> On Thu, Nov 2, 2017 at 7:57 AM, Pedro Larroy <pe...@gmail.com>
> wrote:
>
>> Hi
>>
>> I would like to update the MXNet build instructions.
>>
>> In particular I was thinking that it would be a good idea to update
>> the instructions to use CMake + Ninja. And add more information about
>> the different build flavours.
>>
>>
>> https://mxnet.incubator.apache.org/install/index.html
>>
>>
>> Thoughts?
>>
Re: update build instructions
Posted by Bhavin Thaker <bh...@gmail.com>.
Hi Pedro,
Using Ninja to improve build times is a good suggestion. Can you share the
build times you have observed with and without using Ninja? I presume you
have enabled compile-time options for GPU builds and Distributed MXNet for
the builds you have experimented with.
See also:
https://ninja-build.org/manual.html
Thanks,
Bhavin Thaker.
On Thu, Nov 2, 2017 at 7:57 AM, Pedro Larroy <pe...@gmail.com>
wrote:
> Hi
>
> I would like to update the MXNet build instructions.
>
> In particular I was thinking that it would be a good idea to update
> the instructions to use CMake + Ninja. And add more information about
> the different build flavours.
>
>
> https://mxnet.incubator.apache.org/install/index.html
>
>
> Thoughts?
>