You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by yinxusen <gi...@git.apache.org> on 2016/03/13 03:43:09 UTC
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
GitHub user yinxusen opened a pull request:
https://github.com/apache/spark/pull/11683
[SPARK-13038] [PySpark] Add load/save to pipeline
## What changes were proposed in this pull request?
JIRA issue: https://issues.apache.org/jira/browse/SPARK-13038
1. Add load/save to PySpark Pipeline and PipelineModel
2. Add `_transfer_stage_to_java()` and `_transfer_stage_from_java()` for `JavaWrapper`.
## How was this patch tested?
Test with doctest.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yinxusen/spark SPARK-13038-only
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11683.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11683
----
commit 036529bddf60c2a051136f2813ac4029b16412e1
Author: Xusen Yin <yi...@gmail.com>
Date: 2016-03-11T22:31:49Z
add save/load for pipeline
commit 5c5af9bebb123979168bf29ac3fbe187a094048e
Author: Xusen Yin <yi...@gmail.com>
Date: 2016-03-12T07:14:22Z
a runnable version
commit 81bebfff0e727b11ce93c47edc999b800c43977b
Author: Xusen Yin <yi...@gmail.com>
Date: 2016-03-12T21:57:54Z
add doctest
commit b36bb86bf0c6f21383b0466507435bd19e49c293
Author: Xusen Yin <yi...@gmail.com>
Date: 2016-03-12T22:29:07Z
move stage transfer into wrapper
commit f5d750f35902e2651c1600d4ce848a71bf9df2bd
Author: Xusen Yin <yi...@gmail.com>
Date: 2016-03-13T02:39:28Z
fix corner issues
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197047449
test it please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-195876967
**[Test build #53018 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53018/consoleFull)** for PR 11683 at commit [`f5d750f`](https://github.com/apache/spark/commit/f5d750f35902e2651c1600d4ce848a71bf9df2bd).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56223502
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,87 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
--- End diff --
I'd prefer to move the assertion into `JavaWrapper._transfer_stage_from_java()`, then I can use `isinstance(py_stage, JavaWrapper)` which is more precise.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56222556
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,87 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
--- End diff --
I can see that TrainValidationSplit and its model are also not extending from JavaWrapper. So I'll add them all.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-195858882
**[Test build #53018 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53018/consoleFull)** for PR 11683 at commit [`f5d750f`](https://github.com/apache/spark/commit/f5d750f35902e2651c1600d4ce848a71bf9df2bd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197552039
Hm, if you want to do it under a new JIRA that's OK too. I'll create one now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217493
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,87 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
+ stage.getClass().getName() != "org.apache.spark.PipelineModel",
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ return list(map(JavaWrapper._transfer_stage_from_java, java_stages))
- @abstractmethod
- def _fit(self, dataset):
- """
- Fits a model to the input dataset. This is called by the
- default implementation of fit.
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: fitted model
- """
- raise NotImplementedError()
+def _stages_py2java(py_stages):
+ """
+ Transforms the parameter of Python stages to a Java array of Java stages.
+ :param py_stages: An array of Python stages.
+ :return: A Java array of Java Stages.
+ """
- @since("1.3.0")
- def fit(self, dataset, params=None):
- """
- Fits a model to the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params. If a list/tuple of param maps is given,
- this calls fit on each param map and returns a
- list of models.
- :returns: fitted model(s)
- """
- if params is None:
- params = dict()
- if isinstance(params, (list, tuple)):
- return [self.fit(dataset, paramMap) for paramMap in params]
- elif isinstance(params, dict):
- if params:
- return self.copy(params)._fit(dataset)
- else:
- return self._fit(dataset)
- else:
- raise ValueError("Params must be either a param map or a list/tuple of param maps, "
- "but got %s." % type(params))
+ for stage in py_stages:
+ assert(isinstance(stage, JavaWrapper),
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ gateway = SparkContext._gateway
+ jvm = SparkContext._jvm
+ java_stages = gateway.new_array(jvm.org.apache.spark.ml.PipelineStage, len(py_stages))
+ for idx, stage in enumerate(py_stages):
+ java_stages[idx] = stage._transfer_stage_to_java()
+ return java_stages
@inherit_doc
-class Transformer(Params):
+class _PipelineMLWriter(JavaMLWriter, JavaWrapper):
"""
- Abstract class for transformers that transform one dataset into
- another.
-
- .. versionadded:: 1.3.0
+ Pipeline utility class that can save ML instances through their Scala implementation.
"""
- __metaclass__ = ABCMeta
-
- @abstractmethod
- def _transform(self, dataset):
- """
- Transforms the input dataset.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: transformed dataset
- """
- raise NotImplementedError()
-
- @since("1.3.0")
- def transform(self, dataset, params=None):
- """
- Transforms the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params.
- :returns: transformed dataset
- """
- if params is None:
- params = dict()
- if isinstance(params, dict):
- if params:
- return self.copy(params,)._transform(dataset)
- else:
- return self._transform(dataset)
- else:
- raise ValueError("Params must be either a param map but got %s." % type(params))
+ def __init__(self, instance):
+ self._java_obj = self._new_java_obj("org.apache.spark.ml.Pipeline", instance.uid)
+ jparam = self._java_obj.getParam(instance.stages.name)
+ jstages = _stages_py2java(instance.getStages())
+ self._java_obj.set(jparam.w(jstages))
+ self._jwrite = self._java_obj.write()
@inherit_doc
-class Model(Transformer):
+class _PipelineMLReader(JavaMLReader):
"""
- Abstract class for models that are fitted by estimators.
-
- .. versionadded:: 1.4.0
+ Utility class that can load Pipeline instances through their Scala implementation.
"""
- __metaclass__ = ABCMeta
+ def load(self, path):
+ """Load the Pipeline instance from the input path."""
+ if not isinstance(path, basestring):
+ raise TypeError("path should be a basestring, got type %s" % type(path))
+
+ sc = SparkContext._active_spark_context
+ java_obj = self._jread.load(path)
+ instance = self._clazz()
+ instance._resetUid(java_obj.uid())
+ jparam = java_obj.getParam(instance.stages.name)
+ if java_obj.isDefined(jparam):
+ java_stages = java_obj.getOrDefault(jparam)
+ instance._paramMap[instance.stages] = _stages_java2py(java_stages)
--- End diff --
Call set; do not access paramMap directly
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197010062
Sure.
Cheers
Xusen Yin
> On Mar 15, 2016, at 1:35 PM, jkbradley <no...@github.com> wrote:
>
> One more comment: I just spoke with Xiangrui, and he recommended we not add an underscore to the beginning of private class names (since we don't do that elsewhere). Instead, we can just say that the class is private in the Python docstring.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub <https://github.com/apache/spark/pull/11683#issuecomment-197009204>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217470
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,87 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
--- End diff --
We should also check for CrossValidator and CrossValidatorModel here. Can you please also include them in the error message here and in any message(s) below?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-195876997
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196493904
@mengxr Because I add these imports:
```python
from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
from pyspark.ml.wrapper import JavaWrapper
```
And these imported classes import `Estimator`, `Transformer`, and `Model` simultaneously. It causes the error of cross reference.
And also, I think it's better to split them because the `Estimator`, `Transformer`, and `Model` are abstract classes, while `Pipeline` is not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196653853
**[Test build #53159 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53159/consoleFull)** for PR 11683 at commit [`168bfc1`](https://github.com/apache/spark/commit/168bfc1461c6f089a001d9a31bc0d4f3dbfc4ce5).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217525
--- Diff: python/pyspark/ml/tests.py ---
@@ -145,6 +145,48 @@ def test_pipeline(self):
self.assertEqual(5, transformer3.dataset_index)
self.assertEqual(6, dataset.index)
+ def test_pipeline_presistency(self):
+ sqlContext = SQLContext(self.sc)
+ temp_path = tempfile.mkdtemp()
+
+ try:
+ df = sqlContext.createDataFrame([(["a", "b", "c"],), (["c", "d", "e"],)], ["words"])
+ tf = HashingTF(numFeatures=10, inputCol="words", outputCol="features")
+ pca = PCA(k=2, inputCol="features", outputCol="pca_features")
+ pl = Pipeline(stages=[tf, pca])
+ model = pl.fit(df)
+ pipeline_path = temp_path + "/pipeline"
+ pl.save(pipeline_path)
+ loaded_pipeline = Pipeline.load(pipeline_path)
+ self.assertEqual(loaded_pipeline.uid, pl.uid)
+ self.assertEqual(len(loaded_pipeline.getStages()), 2)
+
+ [loaded_tf, loaded_pca] = loaded_pipeline.getStages()
+ self.assertIsInstance(loaded_tf, HashingTF)
+ self.assertEqual(loaded_tf.uid, tf.uid)
+
+ param = loaded_tf.getParam("numFeatures")
+ self.assertIsInstance(loaded_pca, PCA)
+ self.assertEqual(loaded_tf.getOrDefault(param), tf.getOrDefault(param))
--- End diff --
group checks together; don't mix tf and pca (for clarity)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217505
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -232,6 +203,65 @@ def copy(self, extra=None):
stages = [stage.copy(extra) for stage in that.getStages()]
return that.setStages(stages)
+ @since("2.0.0")
+ def write(self):
+ """Returns an JavaMLWriter instance for this ML instance."""
+ return _PipelineMLWriter(self)
+
+ @since("2.0.0")
+ def save(self, path):
+ """Save this ML instance to the given path, a shortcut of `write().save(path)`."""
+ self.write().save(path)
+
+ @classmethod
+ @since("2.0.0")
+ def read(cls):
+ """Returns an JavaMLReader instance for this class."""
+ return _PipelineMLReader(cls)
+
+ @classmethod
+ @since("2.0.0")
+ def load(cls, path):
+ """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
+ return cls.read().load(path)
+
+
+@inherit_doc
+class PipelineModelMLWriter(JavaMLWriter, JavaWrapper):
--- End diff --
This should be private too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197051289
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53231/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56234912
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,87 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
--- End diff --
Good idea
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-195908350
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53021/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56098420
--- Diff: python/pyspark/ml/base.py ---
@@ -0,0 +1,119 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from abc import ABCMeta, abstractmethod
+
+from pyspark import since
+from pyspark.ml.param import Params
+from pyspark.mllib.common import inherit_doc
+
+
+@inherit_doc
+class Estimator(Params):
+ """
+ Abstract class for estimators that fit models to data.
+ .. versionadded:: 1.3.0
+ """
+
+ __metaclass__ = ABCMeta
+
+ @abstractmethod
+ def _fit(self, dataset):
+ """
+ Fits a model to the input dataset. This is called by the
+ default implementation of fit.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :returns: fitted model
+ """
+ raise NotImplementedError()
+
+ @since("1.3.0")
+ def fit(self, dataset, params=None):
+ """
+ Fits a model to the input dataset with optional parameters.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :param params: an optional param map that overrides embedded
+ params. If a list/tuple of param maps is given,
+ this calls fit on each param map and returns a
+ list of models.
+ :returns: fitted model(s)
+ """
+ if params is None:
+ params = dict()
+ if isinstance(params, (list, tuple)):
+ return [self.fit(dataset, paramMap) for paramMap in params]
+ elif isinstance(params, dict):
+ if params:
+ return self.copy(params)._fit(dataset)
+ else:
+ return self._fit(dataset)
+ else:
+ raise ValueError("Params must be either a param map or a list/tuple of param maps, "
+ "but got %s." % type(params))
+
+
+@inherit_doc
+class Transformer(Params):
+ """
+ Abstract class for transformers that transform one dataset into
+ another.
+ .. versionadded:: 1.3.0
+ """
+
+ __metaclass__ = ABCMeta
+
+ @abstractmethod
+ def _transform(self, dataset):
+ """
+ Transforms the input dataset.
+ :param dataset: input dataset, which is an instance of
--- End diff --
newline. Here and elsewhere, please put a newline between (a) the doc for the class or method and (b) the param doc or any tags (like versionadded) below. Otherwise, the API docs may not generate correctly.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217488
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,87 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
+ stage.getClass().getName() != "org.apache.spark.PipelineModel",
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ return list(map(JavaWrapper._transfer_stage_from_java, java_stages))
- @abstractmethod
- def _fit(self, dataset):
- """
- Fits a model to the input dataset. This is called by the
- default implementation of fit.
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: fitted model
- """
- raise NotImplementedError()
+def _stages_py2java(py_stages):
+ """
+ Transforms the parameter of Python stages to a Java array of Java stages.
+ :param py_stages: An array of Python stages.
+ :return: A Java array of Java Stages.
+ """
- @since("1.3.0")
- def fit(self, dataset, params=None):
- """
- Fits a model to the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params. If a list/tuple of param maps is given,
- this calls fit on each param map and returns a
- list of models.
- :returns: fitted model(s)
- """
- if params is None:
- params = dict()
- if isinstance(params, (list, tuple)):
- return [self.fit(dataset, paramMap) for paramMap in params]
- elif isinstance(params, dict):
- if params:
- return self.copy(params)._fit(dataset)
- else:
- return self._fit(dataset)
- else:
- raise ValueError("Params must be either a param map or a list/tuple of param maps, "
- "but got %s." % type(params))
+ for stage in py_stages:
+ assert(isinstance(stage, JavaWrapper),
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ gateway = SparkContext._gateway
+ jvm = SparkContext._jvm
+ java_stages = gateway.new_array(jvm.org.apache.spark.ml.PipelineStage, len(py_stages))
+ for idx, stage in enumerate(py_stages):
+ java_stages[idx] = stage._transfer_stage_to_java()
+ return java_stages
@inherit_doc
-class Transformer(Params):
+class _PipelineMLWriter(JavaMLWriter, JavaWrapper):
"""
- Abstract class for transformers that transform one dataset into
- another.
-
- .. versionadded:: 1.3.0
+ Pipeline utility class that can save ML instances through their Scala implementation.
"""
- __metaclass__ = ABCMeta
-
- @abstractmethod
- def _transform(self, dataset):
- """
- Transforms the input dataset.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: transformed dataset
- """
- raise NotImplementedError()
-
- @since("1.3.0")
- def transform(self, dataset, params=None):
- """
- Transforms the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params.
- :returns: transformed dataset
- """
- if params is None:
- params = dict()
- if isinstance(params, dict):
- if params:
- return self.copy(params,)._transform(dataset)
- else:
- return self._transform(dataset)
- else:
- raise ValueError("Params must be either a param map but got %s." % type(params))
+ def __init__(self, instance):
+ self._java_obj = self._new_java_obj("org.apache.spark.ml.Pipeline", instance.uid)
+ jparam = self._java_obj.getParam(instance.stages.name)
+ jstages = _stages_py2java(instance.getStages())
+ self._java_obj.set(jparam.w(jstages))
+ self._jwrite = self._java_obj.write()
@inherit_doc
-class Model(Transformer):
+class _PipelineMLReader(JavaMLReader):
"""
- Abstract class for models that are fitted by estimators.
-
- .. versionadded:: 1.4.0
+ Utility class that can load Pipeline instances through their Scala implementation.
"""
- __metaclass__ = ABCMeta
+ def load(self, path):
+ """Load the Pipeline instance from the input path."""
+ if not isinstance(path, basestring):
+ raise TypeError("path should be a basestring, got type %s" % type(path))
+
+ sc = SparkContext._active_spark_context
+ java_obj = self._jread.load(path)
+ instance = self._clazz()
+ instance._resetUid(java_obj.uid())
+ jparam = java_obj.getParam(instance.stages.name)
+ if java_obj.isDefined(jparam):
+ java_stages = java_obj.getOrDefault(jparam)
--- End diff --
Use getStages
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196495577
I think it's better to call save/load from Scala side because we want consistency between Scala/Python implementations. We need to guarantee the model and the pipeline saved by Python/Scala should be loaded via Scala/Python, respectively.
Yes we can implement the pipeline persistence in Python side, but it need more work to keep the consistency.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196489200
I think we have to implement pipeline persistence on the PySpark side because `Pipeline` and other meta algorithms are implemented directly in Python.
Could you comment on why you need to move `Estimator` and `Model` to a different file? It doesn't seem relevant to this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-195858851
@jkbradley @yanboliang
In this PR Pipeline and PipelineModel are kept with Python implementations. However, 2 bad things are
1. It's hard to handle the scenario that Pipeline and PipelineModel themselves are used as stages.
2. I have to add a public PipelineModel constructor to generate Scala PipelineModel in Python side.
I'll close the previous PR: https://github.com/apache/spark/pull/11266
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196104444
**[Test build #53041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53041/consoleFull)** for PR 11683 at commit [`4ce0244`](https://github.com/apache/spark/commit/4ce0244c85a166c6a94f508df0bb694978cc95b7).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56098441
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -146,6 +125,57 @@ class Pipeline(Estimator):
pipeline stages. If there are no stages, the pipeline acts as an
identity transformer.
+ >>> from pyspark.ml.feature import HashingTF
--- End diff --
A unit test this long should go in tests.py, not in a doctest. It'd be nice to have a very short, simple doc test as well though.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11683
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217482
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,87 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
+ stage.getClass().getName() != "org.apache.spark.PipelineModel",
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ return list(map(JavaWrapper._transfer_stage_from_java, java_stages))
- @abstractmethod
- def _fit(self, dataset):
- """
- Fits a model to the input dataset. This is called by the
- default implementation of fit.
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: fitted model
- """
- raise NotImplementedError()
+def _stages_py2java(py_stages):
+ """
+ Transforms the parameter of Python stages to a Java array of Java stages.
+ :param py_stages: An array of Python stages.
+ :return: A Java array of Java Stages.
+ """
- @since("1.3.0")
- def fit(self, dataset, params=None):
- """
- Fits a model to the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params. If a list/tuple of param maps is given,
- this calls fit on each param map and returns a
- list of models.
- :returns: fitted model(s)
- """
- if params is None:
- params = dict()
- if isinstance(params, (list, tuple)):
- return [self.fit(dataset, paramMap) for paramMap in params]
- elif isinstance(params, dict):
- if params:
- return self.copy(params)._fit(dataset)
- else:
- return self._fit(dataset)
- else:
- raise ValueError("Params must be either a param map or a list/tuple of param maps, "
- "but got %s." % type(params))
+ for stage in py_stages:
+ assert(isinstance(stage, JavaWrapper),
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ gateway = SparkContext._gateway
+ jvm = SparkContext._jvm
+ java_stages = gateway.new_array(jvm.org.apache.spark.ml.PipelineStage, len(py_stages))
+ for idx, stage in enumerate(py_stages):
+ java_stages[idx] = stage._transfer_stage_to_java()
+ return java_stages
@inherit_doc
-class Transformer(Params):
+class _PipelineMLWriter(JavaMLWriter, JavaWrapper):
"""
- Abstract class for transformers that transform one dataset into
- another.
-
- .. versionadded:: 1.3.0
+ Pipeline utility class that can save ML instances through their Scala implementation.
"""
- __metaclass__ = ABCMeta
-
- @abstractmethod
- def _transform(self, dataset):
- """
- Transforms the input dataset.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: transformed dataset
- """
- raise NotImplementedError()
-
- @since("1.3.0")
- def transform(self, dataset, params=None):
- """
- Transforms the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params.
- :returns: transformed dataset
- """
- if params is None:
- params = dict()
- if isinstance(params, dict):
- if params:
- return self.copy(params,)._transform(dataset)
- else:
- return self._transform(dataset)
- else:
- raise ValueError("Params must be either a param map but got %s." % type(params))
+ def __init__(self, instance):
+ self._java_obj = self._new_java_obj("org.apache.spark.ml.Pipeline", instance.uid)
+ jparam = self._java_obj.getParam(instance.stages.name)
+ jstages = _stages_py2java(instance.getStages())
+ self._java_obj.set(jparam.w(jstages))
--- End diff --
Can you call ```setStages``` directly instead of getting the Param like this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56246031
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,87 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
+ stage.getClass().getName() != "org.apache.spark.PipelineModel",
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ return list(map(JavaWrapper._transfer_stage_from_java, java_stages))
- @abstractmethod
- def _fit(self, dataset):
- """
- Fits a model to the input dataset. This is called by the
- default implementation of fit.
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: fitted model
- """
- raise NotImplementedError()
+def _stages_py2java(py_stages):
+ """
+ Transforms the parameter of Python stages to a Java array of Java stages.
+ :param py_stages: An array of Python stages.
+ :return: A Java array of Java Stages.
+ """
- @since("1.3.0")
- def fit(self, dataset, params=None):
- """
- Fits a model to the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params. If a list/tuple of param maps is given,
- this calls fit on each param map and returns a
- list of models.
- :returns: fitted model(s)
- """
- if params is None:
- params = dict()
- if isinstance(params, (list, tuple)):
- return [self.fit(dataset, paramMap) for paramMap in params]
- elif isinstance(params, dict):
- if params:
- return self.copy(params)._fit(dataset)
- else:
- return self._fit(dataset)
- else:
- raise ValueError("Params must be either a param map or a list/tuple of param maps, "
- "but got %s." % type(params))
+ for stage in py_stages:
+ assert(isinstance(stage, JavaWrapper),
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ gateway = SparkContext._gateway
+ jvm = SparkContext._jvm
+ java_stages = gateway.new_array(jvm.org.apache.spark.ml.PipelineStage, len(py_stages))
--- End diff --
I prefer to keep using the `new_array`. Because the `ListConverter` return a Java `ArrayList` instead of `Array`, which doesn't fit the Scala `setStages(value: Array[PipelineStage])` interface.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196104554
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53041/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196675963
I'll add follow-up JIRA for nested pipeline support.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-195881298
**[Test build #53021 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53021/consoleFull)** for PR 11683 at commit [`e2d9a81`](https://github.com/apache/spark/commit/e2d9a815bc1fd22ab395a74cb899664a8c7f2758).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196967128
@jkbradley Thanks! I'll fix them soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197009204
One more comment: I just spoke with Xiangrui, and he recommended we not add an underscore to the beginning of private class names (since we don't do that elsewhere). Instead, we can just say that the class is private in the Python docstring.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217518
--- Diff: python/pyspark/ml/tests.py ---
@@ -145,6 +145,48 @@ def test_pipeline(self):
self.assertEqual(5, transformer3.dataset_index)
self.assertEqual(6, dataset.index)
+ def test_pipeline_presistency(self):
--- End diff --
typo: "persistence"
Also, can you please put this test with the other Persistence tests in this file?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197051160
**[Test build #53231 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53231/consoleFull)** for PR 11683 at commit [`3b2db24`](https://github.com/apache/spark/commit/3b2db246981b8f723ba1e285b47d47d080331465).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds the following public classes _(experimental)_:
* `class PipelineMLWriter(JavaMLWriter, JavaWrapper):`
* `class PipelineMLReader(JavaMLReader):`
* `class PipelineModelMLWriter(JavaMLWriter, JavaWrapper):`
* `class PipelineModelMLReader(JavaMLReader):`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196584037
@yinxusen I added some initial comments. I think it looks good overall. However, I do hope to be able to support nested Pipelines, but let's do that in a follow-up PR. Let me know when to take another look. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217512
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -232,6 +203,65 @@ def copy(self, extra=None):
stages = [stage.copy(extra) for stage in that.getStages()]
return that.setStages(stages)
+ @since("2.0.0")
+ def write(self):
+ """Returns an JavaMLWriter instance for this ML instance."""
+ return _PipelineMLWriter(self)
+
+ @since("2.0.0")
+ def save(self, path):
+ """Save this ML instance to the given path, a shortcut of `write().save(path)`."""
+ self.write().save(path)
+
+ @classmethod
+ @since("2.0.0")
+ def read(cls):
+ """Returns an JavaMLReader instance for this class."""
+ return _PipelineMLReader(cls)
+
+ @classmethod
+ @since("2.0.0")
+ def load(cls, path):
+ """Reads an ML instance from the input path, a shortcut of `read().load(path)`."""
+ return cls.read().load(path)
+
+
+@inherit_doc
+class PipelineModelMLWriter(JavaMLWriter, JavaWrapper):
+ """
+ .. note:: Experimental
+
+ PipelineModel utility class that can save ML instances through their Scala implementation.
+
+ .. versionadded:: 2.0.0
+ """
+
+ def __init__(self, instance):
+ self._java_obj = self._new_java_obj("org.apache.spark.ml.PipelineModel",
+ instance.uid,
+ _stages_py2java(instance.stages))
+ self._jwrite = self._java_obj.write()
+
+
+@inherit_doc
+class PipelineModelMLReader(JavaMLReader):
--- End diff --
This should be private too
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196668764
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53159/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196668475
**[Test build #53159 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53159/consoleFull)** for PR 11683 at commit [`168bfc1`](https://github.com/apache/spark/commit/168bfc1461c6f089a001d9a31bc0d4f3dbfc4ce5).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197542597
This LGTM. I'll merge this, but I'll leave the JIRA open for a follow-up PR for handling nested Pipelines. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217459
--- Diff: mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala ---
@@ -297,6 +297,10 @@ class PipelineModel private[ml] (
this(uid, stages.asScala.toArray)
}
+ private[ml] def this(uid: String, stages: Array[PipelineStage]) = {
--- End diff --
I realized that you don't need this. Just use the method right above it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-195908299
**[Test build #53021 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53021/consoleFull)** for PR 11683 at commit [`e2d9a81`](https://github.com/apache/spark/commit/e2d9a815bc1fd22ab395a74cb899664a8c7f2758).
* This patch **fails PySpark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196675833
@jkbradley Ready for another look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196104553
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56098424
--- Diff: python/pyspark/ml/base.py ---
@@ -0,0 +1,119 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from abc import ABCMeta, abstractmethod
+
+from pyspark import since
+from pyspark.ml.param import Params
+from pyspark.mllib.common import inherit_doc
+
+
+@inherit_doc
+class Estimator(Params):
+ """
+ Abstract class for estimators that fit models to data.
+ .. versionadded:: 1.3.0
+ """
+
+ __metaclass__ = ABCMeta
+
+ @abstractmethod
+ def _fit(self, dataset):
+ """
+ Fits a model to the input dataset. This is called by the
+ default implementation of fit.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :returns: fitted model
+ """
+ raise NotImplementedError()
+
+ @since("1.3.0")
+ def fit(self, dataset, params=None):
+ """
+ Fits a model to the input dataset with optional parameters.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :param params: an optional param map that overrides embedded
+ params. If a list/tuple of param maps is given,
+ this calls fit on each param map and returns a
+ list of models.
+ :returns: fitted model(s)
+ """
+ if params is None:
+ params = dict()
+ if isinstance(params, (list, tuple)):
+ return [self.fit(dataset, paramMap) for paramMap in params]
+ elif isinstance(params, dict):
+ if params:
+ return self.copy(params)._fit(dataset)
+ else:
+ return self._fit(dataset)
+ else:
+ raise ValueError("Params must be either a param map or a list/tuple of param maps, "
+ "but got %s." % type(params))
+
+
+@inherit_doc
+class Transformer(Params):
+ """
+ Abstract class for transformers that transform one dataset into
+ another.
+ .. versionadded:: 1.3.0
+ """
+
+ __metaclass__ = ABCMeta
+
+ @abstractmethod
+ def _transform(self, dataset):
+ """
+ Transforms the input dataset.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :returns: transformed dataset
+ """
+ raise NotImplementedError()
+
+ @since("1.3.0")
+ def transform(self, dataset, params=None):
+ """
+ Transforms the input dataset with optional parameters.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :param params: an optional param map that overrides embedded
+ params.
+ :returns: transformed dataset
+ """
+ if params is None:
+ params = dict()
+ if isinstance(params, dict):
+ if params:
+ return self.copy(params,)._transform(dataset)
--- End diff --
remove extra comma (I'm not sure why that was in the original code.)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-195876998
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/53018/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56098425
--- Diff: python/pyspark/ml/base.py ---
@@ -0,0 +1,119 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from abc import ABCMeta, abstractmethod
+
+from pyspark import since
+from pyspark.ml.param import Params
+from pyspark.mllib.common import inherit_doc
+
+
+@inherit_doc
+class Estimator(Params):
+ """
+ Abstract class for estimators that fit models to data.
+ .. versionadded:: 1.3.0
+ """
+
+ __metaclass__ = ABCMeta
+
+ @abstractmethod
+ def _fit(self, dataset):
+ """
+ Fits a model to the input dataset. This is called by the
+ default implementation of fit.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :returns: fitted model
+ """
+ raise NotImplementedError()
+
+ @since("1.3.0")
+ def fit(self, dataset, params=None):
+ """
+ Fits a model to the input dataset with optional parameters.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :param params: an optional param map that overrides embedded
+ params. If a list/tuple of param maps is given,
+ this calls fit on each param map and returns a
+ list of models.
+ :returns: fitted model(s)
+ """
+ if params is None:
+ params = dict()
+ if isinstance(params, (list, tuple)):
+ return [self.fit(dataset, paramMap) for paramMap in params]
+ elif isinstance(params, dict):
+ if params:
+ return self.copy(params)._fit(dataset)
+ else:
+ return self._fit(dataset)
+ else:
+ raise ValueError("Params must be either a param map or a list/tuple of param maps, "
+ "but got %s." % type(params))
+
+
+@inherit_doc
+class Transformer(Params):
+ """
+ Abstract class for transformers that transform one dataset into
+ another.
+ .. versionadded:: 1.3.0
+ """
+
+ __metaclass__ = ABCMeta
+
+ @abstractmethod
+ def _transform(self, dataset):
+ """
+ Transforms the input dataset.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :returns: transformed dataset
+ """
+ raise NotImplementedError()
+
+ @since("1.3.0")
+ def transform(self, dataset, params=None):
+ """
+ Transforms the input dataset with optional parameters.
+ :param dataset: input dataset, which is an instance of
+ :py:class:`pyspark.sql.DataFrame`
+ :param params: an optional param map that overrides embedded
+ params.
+ :returns: transformed dataset
+ """
+ if params is None:
+ params = dict()
+ if isinstance(params, dict):
+ if params:
+ return self.copy(params,)._transform(dataset)
+ else:
+ return self._transform(dataset)
+ else:
+ raise ValueError("Params must be either a param map but got %s." % type(params))
--- End diff --
update doc: remove word "either" (bug in original doc)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197046776
**[Test build #53231 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53231/consoleFull)** for PR 11683 at commit [`3b2db24`](https://github.com/apache/spark/commit/3b2db246981b8f723ba1e285b47d47d080331465).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56098431
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,95 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
+ stage.getClass().getName() != "org.apache.spark.PipelineModel",
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ return list(map(JavaWrapper._transfer_stage_from_java, java_stages))
- @abstractmethod
- def _fit(self, dataset):
- """
- Fits a model to the input dataset. This is called by the
- default implementation of fit.
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: fitted model
- """
- raise NotImplementedError()
+def _stages_py2java(py_stages):
+ """
+ Transforms the parameter of Python stages to a Java array of Java stages.
+ :param py_stages: An array of Python stages.
+ :return: A Java array of Java Stages.
+ """
- @since("1.3.0")
- def fit(self, dataset, params=None):
- """
- Fits a model to the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params. If a list/tuple of param maps is given,
- this calls fit on each param map and returns a
- list of models.
- :returns: fitted model(s)
- """
- if params is None:
- params = dict()
- if isinstance(params, (list, tuple)):
- return [self.fit(dataset, paramMap) for paramMap in params]
- elif isinstance(params, dict):
- if params:
- return self.copy(params)._fit(dataset)
- else:
- return self._fit(dataset)
- else:
- raise ValueError("Params must be either a param map or a list/tuple of param maps, "
- "but got %s." % type(params))
+ for stage in py_stages:
+ assert(isinstance(stage, JavaWrapper),
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ gateway = SparkContext._gateway
+ jvm = SparkContext._jvm
+ java_stages = gateway.new_array(jvm.org.apache.spark.ml.PipelineStage, len(py_stages))
+ for idx, stage in enumerate(py_stages):
+ java_stages[idx] = stage._transfer_stage_to_java()
+ return java_stages
@inherit_doc
-class Transformer(Params):
+class PipelineMLWriter(JavaMLWriter, JavaWrapper):
"""
- Abstract class for transformers that transform one dataset into
- another.
+ .. note:: Experimental
- .. versionadded:: 1.3.0
- """
+ Pipeline utility class that can save ML instances through their Scala implementation.
- __metaclass__ = ABCMeta
-
- @abstractmethod
- def _transform(self, dataset):
- """
- Transforms the input dataset.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: transformed dataset
- """
- raise NotImplementedError()
-
- @since("1.3.0")
- def transform(self, dataset, params=None):
- """
- Transforms the input dataset with optional parameters.
+ .. versionadded:: 2.0.0
+ """
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params.
- :returns: transformed dataset
- """
- if params is None:
- params = dict()
- if isinstance(params, dict):
- if params:
- return self.copy(params,)._transform(dataset)
- else:
- return self._transform(dataset)
- else:
- raise ValueError("Params must be either a param map but got %s." % type(params))
+ def __init__(self, instance):
+ self._java_obj = self._new_java_obj("org.apache.spark.ml.Pipeline", instance.uid)
+ jparam = self._java_obj.getParam(instance.stages.name)
+ jstages = _stages_py2java(instance.getStages())
+ self._java_obj.set(jparam.w(jstages))
+ self._jwrite = self._java_obj.write()
@inherit_doc
-class Model(Transformer):
+class PipelineMLReader(JavaMLReader):
"""
- Abstract class for models that are fitted by estimators.
+ .. note:: Experimental
- .. versionadded:: 1.4.0
+ Utility class that can load Pipeline instances through their Scala implementation.
--- End diff --
Same here; this should be a private API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56098415
--- Diff: mllib/src/main/scala/org/apache/spark/ml/Pipeline.scala ---
@@ -297,6 +297,11 @@ class PipelineModel private[ml] (
this(uid, stages.asScala.toArray)
}
+ @Since("2.0.0")
+ def this(uid: String, stages: Array[PipelineStage]) = {
--- End diff --
This should be private[ml]
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-195908349
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196668758
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197051287
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56098428
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,95 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
+ stage.getClass().getName() != "org.apache.spark.PipelineModel",
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ return list(map(JavaWrapper._transfer_stage_from_java, java_stages))
- @abstractmethod
- def _fit(self, dataset):
- """
- Fits a model to the input dataset. This is called by the
- default implementation of fit.
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: fitted model
- """
- raise NotImplementedError()
+def _stages_py2java(py_stages):
+ """
+ Transforms the parameter of Python stages to a Java array of Java stages.
+ :param py_stages: An array of Python stages.
+ :return: A Java array of Java Stages.
+ """
- @since("1.3.0")
- def fit(self, dataset, params=None):
- """
- Fits a model to the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params. If a list/tuple of param maps is given,
- this calls fit on each param map and returns a
- list of models.
- :returns: fitted model(s)
- """
- if params is None:
- params = dict()
- if isinstance(params, (list, tuple)):
- return [self.fit(dataset, paramMap) for paramMap in params]
- elif isinstance(params, dict):
- if params:
- return self.copy(params)._fit(dataset)
- else:
- return self._fit(dataset)
- else:
- raise ValueError("Params must be either a param map or a list/tuple of param maps, "
- "but got %s." % type(params))
+ for stage in py_stages:
+ assert(isinstance(stage, JavaWrapper),
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ gateway = SparkContext._gateway
+ jvm = SparkContext._jvm
+ java_stages = gateway.new_array(jvm.org.apache.spark.ml.PipelineStage, len(py_stages))
+ for idx, stage in enumerate(py_stages):
+ java_stages[idx] = stage._transfer_stage_to_java()
+ return java_stages
@inherit_doc
-class Transformer(Params):
+class PipelineMLWriter(JavaMLWriter, JavaWrapper):
"""
- Abstract class for transformers that transform one dataset into
- another.
+ .. note:: Experimental
--- End diff --
This should be a private API. Can you please note that this is a private class, and remove the Experimental and versionadded notes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197545649
I'll add new JIRA later.
Sent from my iPhone
> On Mar 16, 2016, at 13:50, jkbradley <no...@github.com> wrote:
>
> This LGTM. I'll merge this, but I'll leave the JIRA open for a follow-up PR for handling nested Pipelines. Thanks!
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly or view it on GitHub
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196563010
I'll take a look at this now. My initial guess is that I agree with @yinxusen about calling save/load from Scala, at least for now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196083710
**[Test build #53041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/53041/consoleFull)** for PR 11683 at commit [`4ce0244`](https://github.com/apache/spark/commit/4ce0244c85a166c6a94f508df0bb694978cc95b7).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on a diff in the pull request:
https://github.com/apache/spark/pull/11683#discussion_r56217476
--- Diff: python/pyspark/ml/pipeline.py ---
@@ -15,116 +15,87 @@
# limitations under the License.
#
-from abc import ABCMeta, abstractmethod
+import sys
+if sys.version > '3':
+ basestring = str
+
+from pyspark import SparkContext
from pyspark import since
+from pyspark.ml import Estimator, Model, Transformer
from pyspark.ml.param import Param, Params
-from pyspark.ml.util import keyword_only
+from pyspark.ml.util import keyword_only, JavaMLWriter, JavaMLReader
+from pyspark.ml.wrapper import JavaWrapper
from pyspark.mllib.common import inherit_doc
-@inherit_doc
-class Estimator(Params):
+def _stages_java2py(java_stages):
"""
- Abstract class for estimators that fit models to data.
-
- .. versionadded:: 1.3.0
+ Transforms the parameter Python stages from a list of Java stages.
+ :param java_stages: An array of Java stages.
+ :return: An array of Python stages.
"""
- __metaclass__ = ABCMeta
+ for stage in java_stages:
+ assert(stage.getClass().getName() != "org.apache.spark.Pipeline" or
+ stage.getClass().getName() != "org.apache.spark.PipelineModel",
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ return list(map(JavaWrapper._transfer_stage_from_java, java_stages))
- @abstractmethod
- def _fit(self, dataset):
- """
- Fits a model to the input dataset. This is called by the
- default implementation of fit.
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :returns: fitted model
- """
- raise NotImplementedError()
+def _stages_py2java(py_stages):
+ """
+ Transforms the parameter of Python stages to a Java array of Java stages.
+ :param py_stages: An array of Python stages.
+ :return: A Java array of Java Stages.
+ """
- @since("1.3.0")
- def fit(self, dataset, params=None):
- """
- Fits a model to the input dataset with optional parameters.
-
- :param dataset: input dataset, which is an instance of
- :py:class:`pyspark.sql.DataFrame`
- :param params: an optional param map that overrides embedded
- params. If a list/tuple of param maps is given,
- this calls fit on each param map and returns a
- list of models.
- :returns: fitted model(s)
- """
- if params is None:
- params = dict()
- if isinstance(params, (list, tuple)):
- return [self.fit(dataset, paramMap) for paramMap in params]
- elif isinstance(params, dict):
- if params:
- return self.copy(params)._fit(dataset)
- else:
- return self._fit(dataset)
- else:
- raise ValueError("Params must be either a param map or a list/tuple of param maps, "
- "but got %s." % type(params))
+ for stage in py_stages:
+ assert(isinstance(stage, JavaWrapper),
+ "Nested Pipeline and PipelineModel are not supported for save/load now.")
+ gateway = SparkContext._gateway
+ jvm = SparkContext._jvm
+ java_stages = gateway.new_array(jvm.org.apache.spark.ml.PipelineStage, len(py_stages))
--- End diff --
This could be shorter if you mimic what's done in pyspark/mllib/common.py:
```
java_stages = map(lambda stage: stage._transfer_stage_to_java(), py_stages)
ListConverter().convert(java_stages, sc._gateway._gateway_client)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196966414
I made a more detailed pass
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197052474
@jkbradley Ready for another look
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by jkbradley <gi...@git.apache.org>.
Github user jkbradley commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-197552521
https://issues.apache.org/jira/browse/SPARK-13951
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13038] [PySpark] Add load/save to pipel...
Posted by yinxusen <gi...@git.apache.org>.
Github user yinxusen commented on the pull request:
https://github.com/apache/spark/pull/11683#issuecomment-196653600
test it please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org