You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2013/12/26 07:31:45 UTC
[28/28] git commit: Merge pull request #283 from tmyklebu/master
Merge pull request #283 from tmyklebu/master
Python bindings for mllib
This pull request contains Python bindings for the regression, clustering, classification, and recommendation tools in mllib.
For each 'train' frontend exposed, there is a Scala stub in PythonMLLibAPI.scala and a Python stub in mllib.py. The Python stub serialises the input RDD and any vector/matrix arguments into a mutually-understood format and calls the Scala stub. The Scala stub deserialises the RDD and the vector/matrix arguments, calls the appropriate 'train' function, serialises the resulting model, and returns the serialised model.
ALSModel is slightly different since a MatrixFactorizationModel has RDDs inside. The Scala stub returns a handle to a Scala MatrixFactorizationModel; prediction is done by calling the Scala predict method.
I have tested these bindings on an x86_64 machine running Linux. There is a risk that these bindings may fail on some choose-your-own-endian platform if Python's endian differs from java.nio.ByteBuffer's idea of the native byte order.
Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/c344ed04
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/c344ed04
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/c344ed04
Branch: refs/heads/master
Commit: c344ed04c7d65d64e87bb50ad6eba57534945398
Parents: 56094bc 9cbcf81
Author: Matei Zaharia <ma...@databricks.com>
Authored: Thu Dec 26 01:31:06 2013 -0500
Committer: Matei Zaharia <ma...@databricks.com>
Committed: Thu Dec 26 01:31:06 2013 -0500
----------------------------------------------------------------------
.../spark/mllib/api/python/PythonMLLibAPI.scala | 232 +++++++++++++++++++
python/pyspark/java_gateway.py | 1 +
python/pyspark/mllib/__init__.py | 20 ++
python/pyspark/mllib/_common.py | 227 ++++++++++++++++++
python/pyspark/mllib/classification.py | 86 +++++++
python/pyspark/mllib/clustering.py | 79 +++++++
python/pyspark/mllib/recommendation.py | 74 ++++++
python/pyspark/mllib/regression.py | 110 +++++++++
python/pyspark/serializers.py | 2 +-
9 files changed, 830 insertions(+), 1 deletion(-)
----------------------------------------------------------------------