configure spark with openblas, thanks

["TheGeorge1918 ." <zh...@gmail.com>]

Hi all,


I’m trying to properly configure OpenBlas in spark ml. I use centos7,
hadoop2.7.2, spark2.0 and python2.7. (I use pyspark to build ml pipeline)


At first I have following warnings


*WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS*

*WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeRefBLAS*


In order to solve the second WARN, I recompile spark from source using


*./build/mvn -Pyarn -Pnetlib-lgpl -Phadoop-$HADOOP_MAJOR_VERSION
-Dhadoop.version=$HADOOP_VERSION -DskipTests clean package*


Now, the second WARN disappeared. To configure openblas, I first installed
openblas and


*ln -s /opt/OpenBLAS/lib/libopenblas.so libblas.so*

*ln -s /opt/OpenBLAS/lib/libopenblas.so liblapack.so*


Still, the warning is there.


I also tried to add


*ln -s /opt/OpenBLAS/lib/libopenblas.so libblas.so.3*

*ln -s /opt/OpenBLAS/lib/libopenblas.so liblapack.so.3*


No luck. It still doesn't work.


Last, I tried to configure numpy to use openblas. Here is the info


*lapack_opt_info:*

*    libraries = ['openblas', 'openblas']*

*    library_dirs = ['/opt/OpenBLAS/lib']*

*    define_macros = [('HAVE_CBLAS', None)]*

*    language = c*

*blas_opt_info:*

*    libraries = ['openblas', 'openblas']*

*    library_dirs = ['/opt/OpenBLAS/lib']*

*    define_macros = [('HAVE_CBLAS', None)]*

*    language = c*

*openblas_info:*

*    libraries = ['openblas', 'openblas']*

*    library_dirs = ['/opt/OpenBLAS/lib']*

*    define_macros = [('HAVE_CBLAS', None)]*

*    language = c*

*openblas_lapack_info:*

*    libraries = ['openblas', 'openblas']*

*    library_dirs = ['/opt/OpenBLAS/lib']*

*    define_macros = [('HAVE_CBLAS', None)]*

*    language = c*

*blas_mkl_info:*

*  NOT AVAILABLE*



I still get the WARN


*WARN BLAS: Failed to load implementation from:
com.github.fommil.netlib.NativeSystemBLAS*



Did I miss something? Thanks a lot.


Best

Xuan