[jira] [Commented] (SPARK-4587) Model export/import

["Peter Prettenhofer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14293892#comment-14293892 ] 

Peter Prettenhofer commented on SPARK-4587:
-------------------------------------------

I read the design document for model import / export

My concern about using parquet as a file format is that it is not widely supported. E.g. it will be hard to deploy a trained mllib pipeline in a Python or go environment. 
To me, a format like thrift or protobuf looks more versatile. E.g. this project by Steven Noble https://github.com/snoble/honeybee contains thrift definitions of decision trees and logistic regression models that I find interesting. My knowledge of parquet is quite limited, it seems that one cannot compare it to thrift serialization and protocol buffers per se but I'm arguing mostly about tooling (ie. the lack of a parquet reader for python/go).

> Model export/import
> -------------------
>
>                 Key: SPARK-4587
>                 URL: https://issues.apache.org/jira/browse/SPARK-4587
>             Project: Spark
>          Issue Type: New Feature
>          Components: ML, MLlib
>            Reporter: Xiangrui Meng
>            Assignee: Joseph K. Bradley
>            Priority: Critical
>
> This is an umbrella JIRA for one of the most requested features on the user mailing list. Model export/import can be done via Java serialization. But it doesn't work for models stored distributively, e.g., ALS and LDA. Ideally, we should provide save/load methods to every model. PMML is an option but it has its limitations. There are couple things we need to discuss: 1) data format, 2) how to preserve partitioning, 3) data compatibility between versions and language APIs, etc.
> UPDATE: [Design doc for model import/export | https://docs.google.com/document/d/1kABFz1ssKJxLGMkboreSl3-I2CdLAOjNh5IQCrnDN3g/edit?usp=sharing]
> This document sketches machine learning model import/export plans, including goals, an API, and development plans.
> The design doc proposes:
> * Support our own Spark-specific format.
> ** This is needed to (a) support distributed models and (b) get model import/export support into Spark quickly (while avoiding new dependencies).
> * Also support PMML
> ** This is needed since it is the only thing approaching an industry standard.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org