You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ma...@apache.org on 2013/12/26 07:31:45 UTC

[28/28] git commit: Merge pull request #283 from tmyklebu/master

Merge pull request #283 from tmyklebu/master

Python bindings for mllib

This pull request contains Python bindings for the regression, clustering, classification, and recommendation tools in mllib.

For each 'train' frontend exposed, there is a Scala stub in PythonMLLibAPI.scala and a Python stub in mllib.py.  The Python stub serialises the input RDD and any vector/matrix arguments into a mutually-understood format and calls the Scala stub.  The Scala stub deserialises the RDD and the vector/matrix arguments, calls the appropriate 'train' function, serialises the resulting model, and returns the serialised model.

ALSModel is slightly different since a MatrixFactorizationModel has RDDs inside.  The Scala stub returns a handle to a Scala MatrixFactorizationModel; prediction is done by calling the Scala predict method.

I have tested these bindings on an x86_64 machine running Linux.  There is a risk that these bindings may fail on some choose-your-own-endian platform if Python's endian differs from java.nio.ByteBuffer's idea of the native byte order.


Project: http://git-wip-us.apache.org/repos/asf/incubator-spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-spark/commit/c344ed04
Tree: http://git-wip-us.apache.org/repos/asf/incubator-spark/tree/c344ed04
Diff: http://git-wip-us.apache.org/repos/asf/incubator-spark/diff/c344ed04

Branch: refs/heads/master
Commit: c344ed04c7d65d64e87bb50ad6eba57534945398
Parents: 56094bc 9cbcf81
Author: Matei Zaharia <ma...@databricks.com>
Authored: Thu Dec 26 01:31:06 2013 -0500
Committer: Matei Zaharia <ma...@databricks.com>
Committed: Thu Dec 26 01:31:06 2013 -0500

----------------------------------------------------------------------
 .../spark/mllib/api/python/PythonMLLibAPI.scala | 232 +++++++++++++++++++
 python/pyspark/java_gateway.py                  |   1 +
 python/pyspark/mllib/__init__.py                |  20 ++
 python/pyspark/mllib/_common.py                 | 227 ++++++++++++++++++
 python/pyspark/mllib/classification.py          |  86 +++++++
 python/pyspark/mllib/clustering.py              |  79 +++++++
 python/pyspark/mllib/recommendation.py          |  74 ++++++
 python/pyspark/mllib/regression.py              | 110 +++++++++
 python/pyspark/serializers.py                   |   2 +-
 9 files changed, 830 insertions(+), 1 deletion(-)
----------------------------------------------------------------------