You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by sr...@apache.org on 2022/09/04 13:30:35 UTC
[spark] branch master updated: [SPARK-40251][MLLIB][DOCS][FOLLOWUP] Update doc for mllib linear algebra acceleration
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 35db3482af6 [SPARK-40251][MLLIB][DOCS][FOLLOWUP] Update doc for mllib linear algebra acceleration
35db3482af6 is described below
commit 35db3482af6a27b303af2b214dd0162b7272bc9d
Author: panbingkun <pb...@gmail.com>
AuthorDate: Sun Sep 4 08:30:20 2022 -0500
[SPARK-40251][MLLIB][DOCS][FOLLOWUP] Update doc for mllib linear algebra acceleration
### What changes were proposed in this pull request?
The pr aims to update doc for mllib linear algebra acceleration after https://github.com/apache/spark/pull/37002 & https://github.com/apache/spark/pull/37700
### Why are the changes needed?
> A.Dependence on netlib-java has been completely eliminated after https://github.com/apache/spark/pull/37002 & https://github.com/apache/spark/pull/37700.
> B.Some interfaces of dev.ludovic.netlib.blas have changed.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
N/A
Closes #37791 from panbingkun/update_doc_for_mllib.
Authored-by: panbingkun <pb...@gmail.com>
Signed-off-by: Sean Owen <sr...@gmail.com>
---
docs/ml-guide.md | 4 ++--
docs/ml-linalg-guide.md | 10 ++++++----
2 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/docs/ml-guide.md b/docs/ml-guide.md
index 3202647240b..572f61ef973 100644
--- a/docs/ml-guide.md
+++ b/docs/ml-guide.md
@@ -62,11 +62,11 @@ The primary Machine Learning API for Spark is now the [DataFrame](sql-programmin
# Dependencies
-MLlib uses linear algebra packages [Breeze](http://www.scalanlp.org/), [dev.ludovic.netlib](https://github.com/luhenry/netlib), and [netlib-java](https://github.com/fommil/netlib-java) for optimised numerical processing[^1]. Those packages may call native acceleration libraries such as [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) or [OpenBLAS](http://www.openblas.net) if they are available as system libraries or in runtime library paths.
+MLlib uses linear algebra packages [Breeze](http://www.scalanlp.org/) and [dev.ludovic.netlib](https://github.com/luhenry/netlib) for optimised numerical processing[^1]. Those packages may call native acceleration libraries such as [Intel MKL](https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html) or [OpenBLAS](http://www.openblas.net) if they are available as system libraries or in runtime library paths.
However, native acceleration libraries can't be distributed with Spark. See [MLlib Linear Algebra Acceleration Guide](ml-linalg-guide.html) for how to enable accelerated linear algebra processing. If accelerated native libraries are not enabled, you will see a warning message like below and a pure JVM implementation will be used instead:
```
-WARN BLAS: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS
+WARNING: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS
```
To use MLlib in Python, you will need [NumPy](http://www.numpy.org) version 1.4 or newer.
diff --git a/docs/ml-linalg-guide.md b/docs/ml-linalg-guide.md
index 1e3d9ccbc82..74dcabf987e 100644
--- a/docs/ml-linalg-guide.md
+++ b/docs/ml-linalg-guide.md
@@ -58,15 +58,17 @@ sudo yum install openblas
To verify native libraries are properly loaded, start `spark-shell` and run the following code:
```
-scala> import dev.ludovic.netlib.NativeBLAS
+scala> import dev.ludovic.netlib.blas.NativeBLAS
scala> NativeBLAS.getInstance()
```
-If they are correctly loaded, it should print `dev.ludovic.netlib.NativeBLAS = dev.ludovic.netlib.blas.JNIBLAS@...`. Otherwise the warnings should be printed:
+If they are correctly loaded, it should print `dev.ludovic.netlib.blas.NativeBLAS = dev.ludovic.netlib.blas.JNIBLAS@...`. Otherwise the warnings should be printed:
```
-WARN NativeBLAS: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS
+WARN InstanceBuilder: Failed to load implementation from:dev.ludovic.netlib.blas.JNIBLAS
+...
java.lang.RuntimeException: Unable to load native implementation
- at dev.ludovic.netlib.NativeBLAS.getInstance(NativeBLAS.java:44)
+ at dev.ludovic.netlib.blas.InstanceBuilder.nativeBlas(InstanceBuilder.java:59)
+ at dev.ludovic.netlib.blas.NativeBLAS.getInstance(NativeBLAS.java:31)
...
```
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org