You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/12/01 16:11:28 UTC

[GitHub] [spark] WeichenXu123 commented on a change in pull request #30471: [SPARK-33520][ML] make CrossValidator/TrainValidateSplit/OneVsRest Reader/Writer support Python backend estimator/model

WeichenXu123 commented on a change in pull request #30471:
URL: https://github.com/apache/spark/pull/30471#discussion_r533535796



##########
File path: python/pyspark/ml/tuning.py
##########
@@ -207,6 +210,205 @@ def _to_java_impl(self):
         return java_estimator, java_epms, java_evaluator
 
 
+class _ValidatorSharedReadWrite:
+
+    @staticmethod
+    def saveImpl(path, instance, sc, extraMetadata=None):
+        from pyspark.ml.classification import OneVsRest
+        numParamsNotJson = 0
+        jsonEstimatorParamMaps = []
+        for paramMap in instance.getEstimatorParamMaps():
+            jsonParamMap = []
+            for p, v in paramMap.items():
+                jsonParam = {'parent': p.parent, 'name': p.name}
+                if (isinstance(v, Estimator) and not (
+                        isinstance(v, _ValidatorParams) or
+                        isinstance(v, OneVsRest))
+                    ) or isinstance(v, Transformer) or \
+                        isinstance(Evaluator):
+                    relative_path = f'epm_{p.name}{numParamsNotJson}'
+                    param_path = os.path.join(path, relative_path)
+                    numParamsNotJson += 1
+                    v.save(param_path)
+                    jsonParam['value'] = relative_path
+                    jsonParam['isJson'] = False
+                elif isinstance(v, MLWritable):
+                    raise RuntimeError(
+                        "ValidatorSharedReadWrite.saveImpl does not handle parameters of type: "
+                        "MLWritable that are not Estimaor/Evaluator/Transformer, and if parameter is estimator,"
+                        "it cannot be Validator or OneVsRest")

Review comment:
       Keep consistent with java side. Java side support `Evaluator` type param. Although current spark estimators don't have such params, but 3rd party estimator may has that.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org