You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by Wenpei <gi...@git.apache.org> on 2016/02/01 09:42:30 UTC
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
GitHub user Wenpei opened a pull request:
https://github.com/apache/spark/pull/11000
[SPARK-13033][ML][PySpark] Add import/export for ml.regression
Add export/import for all estimators and transformers(which have Scala implementation) under pyspark/ml/regression.py.
@yanboliang Please help to review.
For doctest, I though it's enough to add one since it's common usage. But I can add to all if we want it.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Wenpei/spark spark-13033-ml.regression-exprot-import
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11000.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11000
----
commit 32785a8d7116b1c070be44790d4fe028c602024a
Author: Tommy YU <tu...@163.com>
Date: 2016-02-01T08:31:10Z
add import/export for ml.regression
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/11000
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by Wenpei <gi...@git.apache.org>.
Github user Wenpei commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-187047801
@yanboliang I complete this pr, please take a look
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by Wenpei <gi...@git.apache.org>.
Github user Wenpei commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-178970906
Sure, good catch. I have submit a jira 13153 and submit a pr later
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-188216762
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by mengxr <gi...@git.apache.org>.
Github user mengxr commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-189117008
Merged into master. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11000#discussion_r53635702
--- Diff: python/pyspark/ml/regression.py ---
@@ -690,6 +700,18 @@ class AFTSurvivalRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredi
| 0.0|(1,[],[])| 0.0| 1.0|
+-----+---------+------+----------+
...
+ >>> aftsr_path = temp_path + "/aftsr"
+ >>> aftsr.save(aftsr_path)
+ >>> aftsr2 = AFTSurvivalRegression.load(aftsr_path)
+ >>> aftsr2.getMaxIter()
+ 100
+ >>> model_path = temp_path + "/aftsr_model"
+ >>> model.save(model_path)
+ >>> model2 = AFTSurvivalRegressionModel.load(model_path)
+ >>> model.coefficients == model2.coefficients
+ True
+ >>> model.intercept == model2.intercept
+ True
--- End diff --
We should also check equality of model ```scale```.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by Wenpei <gi...@git.apache.org>.
Github user Wenpei commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-178381846
@yanboliang Sorry for last PR that I didn't check scala side.
For regression, there are only three algorithm support MLRead/MLWrite:
LinearRegression
IsotonicRegression
AFTSurvivalRegression
I add export/import api, and doc test currently.
But there is one issues here that doctest failed with below exception. It was caused we didn't set default value for "weightCol" (IsotonicRegression), "quantilesCol"(AFTSurvivalRegression) on scala code side. I add value when constructure to pass doctest, but I thought we should submit a jira for this. How about your idea?
Exception detail.
ir2 = IsotonicRegression.load(ir_path)
Exception raised:
Traceback (most recent call last):
File "C:\Python27\lib\doctest.py", line 1289, in __run
compileflags, 1) in test.globs
File "<doctest __main__.IsotonicRegression[11]>", line 1, in <module>
ir2 = IsotonicRegression.load(ir_path)
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\ml\util.py", line 194, in load
return cls.read().load(path)
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\ml\util.py", line 148, in load
instance._transfer_params_from_java()
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\ml\wrapper.py", line 82, in _tran
fer_params_from_java
value = _java2py(sc, self._java_obj.getOrDefault(java_param))
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\py4j-0.9-src.zip\py4j\java_gateway.py", line 813, in
_call__
answer, self.gateway_client, self.target_id, self.name)
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\pyspark.zip\pyspark\sql\utils.py", line 45, in deco
return f(*a, **kw)
File "C:\aWorkFolder\spark\spark-1.6.0-bin-hadoop2.6\spark-1.6.0-bin-hadoop2.6\python\lib\py4j-0.9-src.zip\py4j\protocol.py", line 308, in get_
eturn_value
format(target_id, ".", name), value)
Py4JJavaError: An error occurred while calling o351.getOrDefault.
: java.util.NoSuchElementException: Failed to find a default value for weightCol
at org.apache.spark.ml.param.Params$$anonfun$getOrDefault$2.apply(params.scala:647)
at org.apache.spark.ml.param.Params$$anonfun$getOrDefault$2.apply(params.scala:647)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.ml.param.Params$class.getOrDefault(params.scala:646)
at org.apache.spark.ml.PipelineStage.getOrDefault(Pipeline.scala:43)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-177855644
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11000#discussion_r53635317
--- Diff: python/pyspark/ml/regression.py ---
@@ -172,6 +172,16 @@ class IsotonicRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredicti
0.0
>>> model.boundaries
DenseVector([0.0, 1.0])
+ >>> ir_path = temp_path + "/ir"
+ >>> ir.save(ir_path)
+ >>> ir2 = IsotonicRegression.load(ir_path)
+ >>> ir2.getIsotonic()
+ True
+ >>> model_path = temp_path + "/ir_model"
+ >>> model.save(model_path)
+ >>> model2 = IsotonicRegressionModel.load(model_path)
+ >>> model.boundaries == model2.boundaries
+ True
--- End diff --
We should also check equality of ```predictions```.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-188191406
**[Test build #51873 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51873/consoleFull)** for PR 11000 at commit [`3646b36`](https://github.com/apache/spark/commit/3646b36f5e90d422f187da110817d9638b6d54bd).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by Wenpei <gi...@git.apache.org>.
Github user Wenpei commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-184131612
@yanboliang OK.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-178575141
@Wenpei It looks like ```_transfer_params_from_java``` did not consider the params which do not have default value and we should handle them. Would you mind to create a jira to track this issue?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by srowen <gi...@git.apache.org>.
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-188189795
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by Wenpei <gi...@git.apache.org>.
Github user Wenpei commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-178953997
Sure, I will submit a jira, I thought we need fix it in scala side that ensure all parameter has default value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11000#discussion_r52871887
--- Diff: python/pyspark/ml/regression.py ---
@@ -179,6 +179,23 @@ class IsotonicRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredicti
0.0
>>> model.boundaries
DenseVector([0.0, 1.0])
+ >>> import os, tempfile
+ >>> path = tempfile.mkdtemp()
+ >>> ir_path = path + "/ir"
+ >>> ir.save(ir_path)
+ >>> ir2 = IsotonicRegression.load(ir_path)
+ >>> ir2.getIsotonic()
+ True
+ >>> model_path = path + "/ir_model"
+ >>> model.save(model_path)
+ >>> model2 = IsotonicRegressionModel.load(model_path)
+ >>> model.boundaries[0] == model2.boundaries[0]
--- End diff --
It's better to check ```model.boundaries == model2.boundaries```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-187221567
Looks good except minor issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-188217094
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by Wenpei <gi...@git.apache.org>.
Github user Wenpei commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-187999921
@mengxr @srowen Can you add me to white list ? or help to launch a jenkins test for this?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11000#discussion_r52872250
--- Diff: python/pyspark/ml/regression.py ---
@@ -697,6 +714,25 @@ class AFTSurvivalRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredi
| 0.0|(1,[],[])| 0.0| 1.0|
+-----+---------+------+----------+
...
+ >>> import os, tempfile
+ >>> path = tempfile.mkdtemp()
+ >>> aftsr_path = path + "/aftsr"
+ >>> aftsr.save(aftsr_path)
+ >>> aftsr2 = AFTSurvivalRegression.load(aftsr_path)
+ >>> aftsr2.getMaxIter()
+ 100
+ >>> model_path = path + "/aftsr_model"
+ >>> model.save(model_path)
+ >>> model2 = AFTSurvivalRegressionModel.load(model_path)
+ >>> model.coefficients[0] == model2.coefficients[0]
--- End diff --
Ditto, because the coefficients is 2-dimension vector in this test case.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-184118656
@Wenpei Please pay attention to the status of #11197 and update this PR corresponding when it get merged.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-188198030
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-188197909
**[Test build #51873 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51873/consoleFull)** for PR 11000 at commit [`3646b36`](https://github.com/apache/spark/commit/3646b36f5e90d422f187da110817d9638b6d54bd).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-188217276
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-187222806
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-188198033
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51873/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-188217429
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-178966434
It should not make all parameters have default value because of some params are not setting default value on purpose. I think we should modify ```_transfer_params_from_java``` to make it not to get the params which do not have default values.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by Wenpei <gi...@git.apache.org>.
Github user Wenpei commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-178975120
I need do a minor change for this PR after 13153 get resoved
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by Wenpei <gi...@git.apache.org>.
Github user Wenpei commented on the pull request:
https://github.com/apache/spark/pull/11000#issuecomment-184090963
@yanboliang @mengxr
Done for this PR. Please help review and launch test.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request: [SPARK-13033][ML][PySpark] Add import/export f...
Posted by yanboliang <gi...@git.apache.org>.
Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/11000#discussion_r51400507
--- Diff: python/pyspark/ml/regression.py ---
@@ -447,7 +447,7 @@ def _create_model(self, java_model):
@inherit_doc
-class DecisionTreeModel(JavaModel):
+class DecisionTreeModel(JavaModel, MLWritable, MLReadable):
"""Abstraction for Decision Tree models.
--- End diff --
@Wenpei Please check wether it supports ```save/load``` for the peer Scala implementation. Some algorithms such as ```DecisionTree``` did not support it currently. And you should add doc test that will test the correctness of your modification.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org