You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@mxnet.apache.org by GitBox <gi...@apache.org> on 2017/10/25 22:42:27 UTC

[GitHub] Garglesoap opened a new issue #8429: R package for GPU mxnet does not install on amazon linux AMI

Garglesoap opened a new issue #8429: R package for GPU mxnet does not install on amazon linux AMI
URL: https://github.com/apache/incubator-mxnet/issues/8429
 
 
   I am having trouble installing mxnet GPU for R on Amazon deep learning linux AMI. The environment variables are such a mess that it?s a nightmare for any non-expert sys-admin to figure out. 
   
   Step 1: install the ridiculous amount of missing/broken programs and R packages
   sudo yum install R
   sudo yum install libxml2-devel   
   sudo yum install cairo-devel
   sudo yum install giflib-devel
   sudo yum install libXt-devel
   sudo R
   install.packages("devtools")
   library(devtools)
   install_github("igraph/rigraph")
   install.packages(?DiagrammeR?) 
   install.packages(?roxygen2?)
   install.packages(?rgexf?)
   install.packages(?influenceR?)
   install.packages(?Cairo?)
   install.packages(?imager?)
   
   Step 2: edit the config.mk file
   cd /src/mxnet
   cp make/config.mk .
   echo "USE_BLAS=openblas" >>config.mk
   echo "ADD_CFLAGS += -I/usr/include/openblas" >>config.mk
   echo "ADD_LDFLAGS += -lopencv_core -lopencv_imgproc -lopencv_imgcodecs" >>config.mk
   echo "USE_CUDA=1" >>config.mk
   echo "USE_CUDA_PATH=/usr/local/cuda" >>config.mk
   echo "USE_CUDNN=1" >>config.mk
   
   *note even though the USE_CUDA_PATH is set, it STILL cannot find libcudart.so and needs to be linked in the make command (shown later)
   
   Step 3: make new config file so make command can find libcudart.so
   /etc/ld.so.conf.d/cuda.conf
   	? /usr/local/cuda-8.0/lib64
   sudo ldconfig
   * note this was posted by nvidia but does absolutely nothing to help the make rpkg
   
   Step 4: set up R directories 
   Rscript -e "install.packages('devtools', repo = 'https://cran.rstudio.com')"
   cd R-package
   Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cran.rstudio.com'));
   install_deps(dependencies = TRUE)"
   cd ..
   
   Step 5: make 
   cd /src/mxnet
   sudo make -j8
   
   Result: 
   make CXX=g++ DEPS_PATH=/home/ec2-user/src/mxnet/deps -C /home/ec2-user/src/mxnet/ps-lite ps
   cd /home/ec2-user/src/mxnet/dmlc-core; make libdmlc.a USE_SSE=1 config=/home/ec2-user/src/mxnet/config.mk; cd /home/ec2-user/src/mxnet
   make[1]: Entering directory `/home/ec2-user/src/mxnet/dmlc-core'
   make[1]: `libdmlc.a' is up to date.
   make[1]: Leaving directory `/home/ec2-user/src/mxnet/dmlc-core'
   make[1]: Entering directory `/home/ec2-user/src/mxnet/ps-lite'
   make[1]: Nothing to be done for `ps'.
   make[1]: Leaving directory `/home/ec2-user/src/mxnet/ps-lite'
   ar crv lib/libmxnet.a
   *note, even when changing the config.mk file, the make command always returns ?nothing to update?
   
   Step 6: attempt to make rpkg
   Cd /src/mxnet
   Sudo make rpkg
   
   Error: 
   Error: package or namespace load failed for ?mxnet?:
    .onLoad failed in loadNamespace() for 'mxnet', details:
     call: dyn.load(file, DLLpath = DLLpath, ...)
     error: unable to load shared object '/usr/lib64/R/library/mxnet/libs/libmxnet.so':
     libcudart.so.8.0: cannot open shared object file: No such file or directory
   Error: loading failed
   Execution halted
   ERROR: loading failed
   
   So it?s looking in a location that doesn?t exist: /usr/lib64/R/library/mxnet/libs/
   When the file actually lives: 
   /home/ec2-user/src/mxnet/R-package/inst/libs/libmxnet.so
   or
   /home/ec2-user/src/mxnet/lib/libmxnet.so
   
   
   What I?ve tried so far:
   sudo LD_LIBRARY_PATH=/usr/local/cuda/lib64 make rpkg
   This will fix the missing libcudart.so.8.0 issue but it is simply replace with:
   libmklml_intel.so: cannot open shared object file: No such file or directory
    as well as the original ?cannot find libmxnet.so 
   
   Also tried:
   1. actually creating directories (/usr/lib64/R/library/mxnet/libs/) and then copying libmxnet.so there
   Result: same error
   2. adding /home/ec2-user/src/mxnet/R-package/inst/libs/ to the make command
   sudo LD_LIBRARY_PATH=/home/ec2-user/src/mxnet/R-package/inst/libs make rpkg
   Result: same error
   3. a ridiculous amount of environment labels all of which failed:
   export MXNET_HOME=/usr/lib64/R/library/mxnet/libs/
   export MXNET_HOME=/usr/lib64/R/library/mxnet/libs/libmxnet.so              
   sudo ldconfig /usr/local/cuda/lib64
   sudo ln -s /usr/lib64/R/library/mxnet/libs /usr/lib
   sudo ln -s /usr/lib64/R/library/mxnet/libs/libmxnet.so /usr/lib
   sudo ln -s /usr/local/lib/libmklml_intel.so /usr/lib
   sudo ln -s /usr/local/lib/libiomp5.so /usr/lib
   sudo ln -s /usr/local /usr/lib
   export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64/libcudart.so.8.0
   export LD_LIBRARY_PATH=/usr/lib64/R/library/mxnet/libs/libmxnet.so /usr/lib
   export LD_LIBRARY_PATH=/usr/local/cuda-8.0/targets/x86_64-linux/lib/:$LD_LIBRARY_PATH
   export LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64/libcudart.so.8.0
   
   In all, ONE of these worked, because I briefly got mxnet R package working before it fell apart again. I?ve dropped 50+ hours into this installation, which, frankly is ridiculous.
   
   I don?t have 5+ years of linux sys admin knowledge so if you?d like please be a bit more helpful then ?fix environment variables.? I can tell that?s obviously what?s wrong yet have no idea what ?fix environment variables? entails. 
   
   To top it off, even after successful install of the R package, it STILL won?t work until setting Rstudio server?s config file to: rsession-ld-library-path=/opt/local/lib:/usr/local/cuda/lib64
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services