You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "wbo4958 (via GitHub)" <gi...@apache.org> on 2024/11/07 10:16:10 UTC
[PR] [SPARK-49907] Support spark.ml on Connect [spark]
wbo4958 opened a new pull request, #48791:
URL: https://github.com/apache/spark/pull/48791
### What changes were proposed in this pull request?
This PR supports running spark.ml on connect
### Why are the changes needed?
It's a new feature that makes spark.ml run on connect environment.
### Does this PR introduce _any_ user-facing change?
Yes, new feature.
### How was this patch tested?
The below manual test can work without any exception.
``` python
(pyspark) user@bobby:~ $ pyspark --remote sc://localhost
Python 3.11.10 (main, Oct 3 2024, 07:29:13) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 4.0.0.dev0
/_/
Using Python version 3.11.10 (main, Oct 3 2024 07:29:13)
Client connected to the Spark Connect server at localhost
SparkSession available as 'spark'.
>>> from pyspark.ml.classification import (LogisticRegression,
... LogisticRegressionModel)
>>> from pyspark.ml.linalg import Vectors
>>>
>>> df = spark.createDataFrame([
... (Vectors.dense([1.0, 2.0]), 1),
... (Vectors.dense([2.0, -1.0]), 1),
... (Vectors.dense([-3.0, -2.0]), 0),
... (Vectors.dense([-1.0, -2.0]), 0),
... ], schema=['features', 'label'])
>>> lr = LogisticRegression()
>>> lr.setMaxIter(30)
LogisticRegression_a842693fc5e7
>>> model: LogisticRegressionModel = lr.fit(df)
>>> model.predictRaw(Vectors.dense([1.0, 2.0]))
DenseVector([-21.1048, 21.1048])
>>> assert model.getMaxIter() == 30
>>> model.summary.roc.show()
+---+---+
|FPR|TPR|
+---+---+
|0.0|0.0|
|0.0|0.5|
|0.0|1.0|
|0.5|1.0|
|1.0|1.0|
|1.0|1.0|
+---+---+
>>> model.summary.weightedRecall
1.0
>>> model.summary.recallByLabel
[1.0, 1.0]
>>> model.coefficients
DenseVector([10.3964, 4.513])
>>> model.intercept
1.6823489096339976
>>> model.transform(df).show()
+-----------+-----+--------------------+--------------------+----------+
| features|label| rawPrediction| probability|prediction|
+-----------+-----+--------------------+--------------------+----------+
| [1.0,2.0]| 1|[-21.104818251026...|[6.82800596288997...| 1.0|
| [2.0,-1.0]| 1|[-17.962094978515...|[1.58183529116629...| 1.0|
|[-3.0,-2.0]| 0|[38.5329050234205...| [1.0,0.0]| 0.0|
|[-1.0,-2.0]| 0|[17.7401204317582...|[0.99999998025016...| 0.0|
+-----------+-----+--------------------+--------------------+----------+
>>> model.write().overwrite().save("/tmp/connect-ml-demo")
>>> loaded_model = LogisticRegressionModel.load("/tmp/connect-ml-demo")
>>> assert loaded_model.getMaxIter() == 30
>>> loaded_model.transform(df).show()
+-----------+-----+--------------------+--------------------+----------+
| features|label| rawPrediction| probability|prediction|
+-----------+-----+--------------------+--------------------+----------+
| [1.0,2.0]| 1|[-21.104818251026...|[6.82800596288997...| 1.0|
| [2.0,-1.0]| 1|[-17.962094978515...|[1.58183529116629...| 1.0|
|[-3.0,-2.0]| 0|[38.5329050234205...| [1.0,0.0]| 0.0|
|[-1.0,-2.0]| 0|[17.7401204317582...|[0.99999998025016...| 0.0|
+-----------+-----+--------------------+--------------------+----------+
```
### Was this patch authored or co-authored using generative AI tooling?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2487967686
I\ll start reviewing. High-level question: On the PR description you say:
> Read/write operations for LogisticRegression
> Evaluation of LogisticRegressionModel on a dataset
What does this mean? What's missing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1908550792
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
Review Comment:
The benefit of the ServiceLoader is that it forces the class to be in a specific interface and providers have to declare the classes that they expose properly. This makes it a very good contract and not just loading arbitrary classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2499883456
Hi @wbo4958 ,
I just applied the following patch on your PR:
```
diff --git a/mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Estimator b/mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Estimator
index f3fd21ad4c3..fe690cb8eeb 100644
--- a/mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Estimator
+++ b/mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Estimator
@@ -17,4 +17,13 @@
# Spark Connect ML uses ServiceLoader to find out the supported Spark Ml estimators.
# So register the supported estimator here if you're trying to add a new one.
-org.apache.spark.ml.classification.LogisticRegression
\ No newline at end of file
+org.apache.spark.ml.classification.LogisticRegression
+org.apache.spark.ml.feature.QuantileDiscretizer
+org.apache.spark.ml.feature.StringIndexer
+org.apache.spark.ml.feature.OneHotEncoder
+org.apache.spark.ml.feature.PCA
+org.apache.spark.ml.feature.StandardScaler
+org.apache.spark.ml.feature.MaxAbsScaler
+org.apache.spark.ml.feature.MinMaxScaler
+org.apache.spark.ml.feature.VectorIndexer
+org.apache.spark.ml.feature.RobustScaler
\ No newline at end of file
diff --git a/mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Transformer b/mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Transformer
index 24b1133cb5b..da5db5834e7 100644
--- a/mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Transformer
+++ b/mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Transformer
@@ -17,4 +17,17 @@
# Spark Connect ML uses ServiceLoader to find out the supported Spark Ml non-model transformer.
# So register the supported transformer here if you're trying to add a new one.
-org.apache.spark.ml.feature.VectorAssembler
\ No newline at end of file
+org.apache.spark.ml.feature.VectorAssembler
+org.apache.spark.ml.feature.Bucketizer
+org.apache.spark.ml.feature.Tokenizer
+org.apache.spark.ml.feature.RegexTokenizer
+org.apache.spark.ml.feature.StopWordsRemover
+org.apache.spark.ml.feature.NGram
+org.apache.spark.ml.feature.Binarizer
+org.apache.spark.ml.feature.Normalizer
+org.apache.spark.ml.feature.PolynomialExpansion
+org.apache.spark.ml.feature.DCT
+org.apache.spark.ml.feature.Interaction
+org.apache.spark.ml.feature.ElementwiseProduct
+org.apache.spark.ml.feature.SQLTransformer
+org.apache.spark.ml.feature.VectorSizeHint
```
To enable all of the feature engineering algorithms and it all worked out of the box modulo very few:
* StopWordRemove tries to detect the locale based on a Java String which obviously doesn't work
* StringIndexer fails with:
`pyspark.errors.exceptions.connect.SparkConnectGrpcException: (java.lang.NullPointerException) Cannot invoke "org.apache.spark.ml.Model.copy(org.apache.spark.ml.param.ParamMap)" because the return value of "org.apache.spark.sql.connect.ml.ModelAttributeHelper.model()" is null`
* Normlizer tries to copy a java object ` File "/Users/martin.grund/Development/spark/python/pyspark/ml/wrapper.py", line 357, in copy
that._java_obj = self._java_obj.copy(self._empty_java_param_map())`
* QunatileDiscretizer tries to call another Java method -> ` File "/Users/martin.grund/Development/spark/python/pyspark/ml/feature.py", line 3756, in _create_model
splits=list(java_model.getSplits()),`
Otherwise I could run all of the examples from the Spark Homepage! Awesome!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876703045
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
Review Comment:
Not a big problem. This prevents users from plugging an ML library through either addJar or addArtifact later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "rouroufulcat (via GitHub)" <gi...@apache.org>.
rouroufulcat commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2530715195
Hello. I suppose most of the proto and server part of this pr are almost the same as that in https://github.com/apache/spark/pull/40479. So maybe at least this pr#40479 should be mentioned or at least the author should be added as coauthor?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876944940
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Model, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw new RuntimeException("Unsupported parameter type")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the fit command
+ *
+ * @param fit
+ * command
+ * @return
+ * an Estimator
+ */
+ def getEstimator(fit: proto.MlCommand.Fit): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the fit command
+ val name = fit.getEstimator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw new RuntimeException(s"Failed to find estimator: $name")
+ }
+ val uid = fit.getEstimator.getUid
+ val estimator: Estimator[_] = estimators(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Estimator[_]]
+
+ // Set parameters for the estimator
+ val params = fit.getParams
+ MLUtils.setInstanceParams(estimator, params)
+ estimator
+ }
+
+ def loadModel(className: String, path: String): Model[_] = {
+ // scalastyle:off classforname
+ val clazz = Class.forName(className)
+ // scalastyle:on classforname
+ val model = invokeStaticMethod(clazz, "load", path)
+ model.asInstanceOf[Model[_]]
+ }
+
+ /**
+ * Get the transformer instance according to the transform proto
+ *
+ * @param transformProto
+ * transform proto
+ * @return
+ * a Transformer
+ */
+ def getTransformer(transformProto: proto.MlRelation.Transform): Transformer = {
+ // Get the transformer name
+ val name = transformProto.getTransformer.getName
+ if (transformers.isEmpty || !transformers.contains(name)) {
+ throw new RuntimeException(s"Failed to find transformer: $name")
+ }
+ val uid = transformProto.getTransformer.getUid
+ val transformer = transformers(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Transformer]
+
+ val params = transformProto.getParams
+ MLUtils.setInstanceParams(transformer, params)
+ transformer
+ }
+
+ private lazy val ALLOWED_ATTRIBUTES = HashSet(
Review Comment:
Is it an option to push this into the individual classes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1886000855
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
Review Comment:
I'm wondering if I should move the commands and result-of-commands from ml.proto to command.proto and base.proto, so after that, there will be only ml_common.proto left.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1868685572
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ * @param sessionHolder
+ * @param objIdentifier
+ * @param method
+ * @param argValues
+ * @param argClasses
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ protected lazy val instance = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(instance, lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+}
+
+private class ModelAttributeHelper(
+ sessionHolder: SessionHolder,
+ objIdentifier: String,
+ method: Option[String],
+ argValues: Array[Object] = Array.empty,
+ argClasses: Array[Class[_]] = Array.empty)
+ extends AttributeHelper(sessionHolder, objIdentifier, method, argValues, argClasses) {
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val model = instance.asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object AttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): AttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new AttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+// MLHandler is a utility to group all ML operations
+object MLHandler extends Logging {
+ def handleMlCommand(
+ sessionHolder: SessionHolder,
+ mlCommand: proto.MlCommand): proto.MlCommandResult = {
+
+ val mlCache = sessionHolder.mlCache
+
+ mlCommand.getCommandCase match {
+ case proto.MlCommand.CommandCase.FIT =>
+ val fitCmd = mlCommand.getFit
+ val estimatorProto = fitCmd.getEstimator
+ assert(estimatorProto.getType == proto.MlOperator.OperatorType.ESTIMATOR)
+
+ val dataset = MLUtils.parseRelationProto(fitCmd.getDataset, sessionHolder)
+ val estimator = MLUtils.getEstimator(estimatorProto, Some(fitCmd.getParams))
+ val model = estimator.fit(dataset).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id)))
+ .build()
+
+ case proto.MlCommand.CommandCase.FETCH_ATTR =>
+ val args = mlCommand.getFetchAttr.getArgsList.asScala.toArray
+ val helper = AttributeHelper(
+ sessionHolder,
+ mlCommand.getFetchAttr.getObjRef.getId,
+ Option(mlCommand.getFetchAttr.getMethod),
+ args)
+ val attrResult = helper.getAttribute
+ attrResult match {
+ case s: Summary =>
+ val id = mlCache.register(s)
+ proto.MlCommandResult.newBuilder().setSummary(id).build()
+ case _ =>
+ val param = Serializer.serializeParam(attrResult)
+ proto.MlCommandResult.newBuilder().setParam(param).build()
+ }
+
+ case proto.MlCommand.CommandCase.DELETE =>
+ val modelId = mlCommand.getDelete.getObjRef.getId
+ var result = false
+ if (!modelId.contains(".")) {
+ mlCache.remove(modelId)
+ result = true
+ }
+ proto.MlCommandResult
+ .newBuilder()
+ .setParam(
+ proto.Param
+ .newBuilder()
+ .setLiteral(LiteralValueProtoConverter.toLiteralProto(result))
+ .build())
+ .build()
+
+ case proto.MlCommand.CommandCase.WRITE =>
+ mlCommand.getWrite.getTypeCase match {
+ case proto.MlCommand.Writer.TypeCase.OBJ_REF => // save a model
+ val modelId = mlCommand.getWrite.getObjRef.getId
+ val model = mlCache.get(modelId).asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, mlCommand.getWrite.getParams)
+
+ copiedModel match {
+ case m: MLWritable => MLUtils.write(m, mlCommand.getWrite)
+ case _ => throw new RuntimeException("Failed to handle model.save")
Review Comment:
I think this is my last bigger comment, I would kindly ask to avoid using RuntimeExceptions and instead create a new MLException in SparkConnect that inherits from SparkException and uses the proper message templates.
What you need for that:
- Use XX000 as the error code https://github.com/apache/spark/blob/master/common/utils/src/main/resources/error/error-states.json#L7474
- create a spark connect ML related sub class similar to asof join https://github.com/apache/spark/blob/master/common/utils/src/main/resources/error/error-conditions.json#L85-L107 the error classes must be ordered alphabetically :)
- use the sqlstate from above
- create new sub classes for the error here with proper message templates etc. (cf. https://github.com/apache/spark/blob/13315eeec07e2aebcc05dfee762bbd060ae192ec/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/joinTypes.scala#L148-L153)
Please do this for all exceptions used here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "martin-g (via GitHub)" <gi...@apache.org>.
martin-g commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2576809670
I guess you pinged me by mistake.
I am not familiar with this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1915986310
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
Review Comment:
Ya, it looks like that. I made a follow-PR. Thank you, @LuciferYang .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2473809476
@michTalebzadeh the idea is that we give Spark Connect users the same functionality as existing classic users.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907] Support spark.ml on Connect [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2475061251
cc @WeichenXu123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876933726
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the operator name
+ val name = operator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported Estimator, found ${name}")
+ }
+ val uid = operator.getUid
+ val estimator: Estimator[_] = estimators(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Estimator[_]]
+
+ // Set parameters for the estimator
+ params.foreach(p => MLUtils.setInstanceParams(estimator, p))
+ estimator
+ }
+
+ def load(className: String, path: String): Object = {
+ // scalastyle:off classforname
+ val clazz = Class.forName(className)
Review Comment:
Use `SparkClassUtils.classForName(..)`.
Also if this is Scala, you can also resolve the Module class and call load on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876717230
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLCache.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.ml
+
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.internal.Logging
+
+/**
+ * MLCache is for caching ML objects, typically for models and summaries evaluated by a model.
+ */
+private[connect] class MLCache extends Logging {
+ private val cachedModel: ConcurrentHashMap[String, Object] =
+ new ConcurrentHashMap[String, Object]()
+
+ /**
+ * Cache an object into a map of MLCache, and return its key
+ * @param obj
+ * the object to be cached
+ * @return
+ * the key
+ */
+ def register(obj: Object): String = {
+ val objectId = UUID.randomUUID().toString.takeRight(12)
Review Comment:
Why only 12 characters? This is 44 bits of entropy (11 characters, 1 character is constant..., 4 bits per character)... Is that enough to avoid collisions? An alternative would be to generate a predictable Id.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2497632688
> I\ll start reviewing. High-level question: On the PR description you say:
>
> > Read/write operations for LogisticRegression
> > Evaluation of LogisticRegressionModel on a dataset
>
> What does this mean? What's missing?
Now, the new commit has supported `Read/write operations for LogisticRegression`
> Evaluation of LogisticRegressionModel on a dataset
The [evaluate](https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/classification/LogisticRegression.scala#L1197) returns a Summary which is not attaching to any model, so we should not retrieve the attribute of the summary according to the chain of `model` -> `summary` -> `attribute`. Instead, we should cache the summary just like the model, and then retrieve it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2500950542
@wbo4958 we shouldn't do all in one PR, I was mostly curious what the delta effort is an how hard it's going to be to add new algorithms.
Since most of the work for adding new estimators and transformers is going to be testing, I think its perfect to do it in a follow up PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1886076892
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
Review Comment:
Done. Thx
##########
sql/connect/common/src/main/protobuf/spark/connect/ml_common.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
Review Comment:
Done. Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876957116
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
+ oneof command {
Review Comment:
It would be good to explain what these commands actually do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1878874332
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
Review Comment:
Can we do this at the protocol level?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876985223
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
+ oneof command {
+ Fit fit = 1;
+ FetchAttr fetch_attr = 2;
+ Delete delete = 3;
+ Writer write = 4;
+ Reader read = 5;
Review Comment:
Use `Read` instead of `Reader`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip closed pull request #48791: [SPARK-49907][ML][CONNECT] Support spark.ml on Connect
URL: https://github.com/apache/spark/pull/48791
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2500035070
I did play some more with different estimators and LinearRegression and RandomForrestRegressor worked nicely as well. I ran into some issues with the training summary and getting some of the attributes back but that was caught properly.
The generic base class of `Evaluator` is not yet supported though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1856319287
##########
python/pyspark/ml/remote/readwrite.py:
##########
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import cast, Type, TYPE_CHECKING
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, deserialize, deserialize_param
+from pyspark.ml.util import MLWriter, MLReader, RL
+from pyspark.ml.wrapper import JavaWrapper
+
+if TYPE_CHECKING:
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+ from pyspark.core.context import SparkContext
+
+
+class RemoteMLWriter(MLWriter):
+ def __init__(self, instance: "JavaMLWritable") -> None:
+ super().__init__()
+ self._instance = instance
+
+ @property
+ def sc(self) -> "SparkContext":
+ raise RuntimeError("Accessing SparkContext is not supported on Connect")
+
+ def save(self, path: str) -> None:
+ from pyspark.ml.wrapper import JavaModel
+
+ if isinstance(self._instance, JavaModel):
+ from pyspark.sql.connect.session import SparkSession
+
+ session = SparkSession.getActiveSession()
+ assert session is not None
+ instance = cast("JavaModel", self._instance)
+ params = serialize_ml_params(instance, session.client)
+
+ assert isinstance(instance._java_obj, str)
+ writer = pb2.MlCommand.Writer(
+ model_ref=pb2.ModelRef(id=instance._java_obj),
+ params=params,
+ path=path,
+ should_overwrite=self.shouldOverwrite,
+ options=self.optionMap,
+ )
+ req = session.client._execute_plan_request_with_metadata()
+ req.plan.ml_command.write.CopyFrom(writer)
+ session.client.execute_ml(req)
+
+
+class RemoteMLReader(MLReader[RL]):
+ def __init__(self, clazz: Type["JavaMLReadable[RL]"]) -> None:
+ super().__init__()
+ self._clazz = clazz
Review Comment:
Yes, true. The estimator and model are extending from JavaMLReadable. so it's generalizable.
##########
python/pyspark/ml/remote/proto.py:
##########
@@ -0,0 +1,86 @@
+#
Review Comment:
I could also move the existing "remote" to the connect if necessary.
##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -384,6 +386,9 @@ message ExecutePlanResponse {
// Response for command that checkpoints a DataFrame.
CheckpointCommandResult checkpoint_command_result = 19;
+ // ML command response
+ MlCommandResponse ml_command_result = 100;
Review Comment:
It's already there.
##########
mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Estimator:
##########
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.spark.ml.classification.LogisticRegression
Review Comment:
sounds good. Added the comment here
##########
python/pyspark/ml/remote/serialize.py:
##########
@@ -0,0 +1,120 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from typing import Any, List, TYPE_CHECKING, Mapping, Optional
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.linalg import (
+ Vectors,
+ Matrices,
+ DenseVector,
+ SparseVector,
+ DenseMatrix,
+ SparseMatrix,
+)
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+from pyspark.sql.connect.expressions import LiteralExpression
+
+if TYPE_CHECKING:
+ from pyspark.sql.connect.client import SparkConnectClient
+ from pyspark.ml.param import Params
+
+
+def serialize_param(value: Any, client: "SparkConnectClient") -> pb2.Param:
+ if isinstance(value, DenseVector):
+ return pb2.Param(vector=pb2.Vector(dense=pb2.Vector.Dense(value=value.values.tolist())))
+ elif isinstance(value, SparseVector):
+ return pb2.Param(
+ vector=pb2.Vector(
+ sparse=pb2.Vector.Sparse(
+ size=value.size, index=value.indices.tolist(), value=value.values.tolist()
+ )
+ )
+ )
+ elif isinstance(value, DenseMatrix):
+ return pb2.Param(
+ matrix=pb2.Matrix(
+ dense=pb2.Matrix.Dense(
+ num_rows=value.numRows, num_cols=value.numCols, value=value.values.tolist()
+ )
+ )
+ )
+ elif isinstance(value, SparseMatrix):
+ return pb2.Param(
+ matrix=pb2.Matrix(
+ sparse=pb2.Matrix.Sparse(
+ num_rows=value.numRows,
+ num_cols=value.numCols,
+ colptr=value.colPtrs.tolist(),
+ row_index=value.rowIndices.tolist(),
+ value=value.values.tolist(),
+ )
+ )
+ )
+ else:
+ literal = LiteralExpression._from_value(value).to_plan(client).literal
+ return pb2.Param(literal=literal)
+
+
+def serialize(client: "SparkConnectClient", *args: Any) -> List[Any]:
+ result = []
+ for arg in args:
+ if isinstance(arg, RemoteDataFrame):
+ result.append(pb2.FetchModelAttr.Args(input=arg._plan.plan(client)))
+ else:
+ result.append(pb2.FetchModelAttr.Args(param=serialize_param(arg, client)))
+ return result
+
+
+def deserialize_param(param: pb2.Param) -> Any:
+ if param.HasField("literal"):
+ return LiteralExpression._to_value(param.literal)
+ if param.HasField("vector"):
+ vector = param.vector
+ # TODO support sparse vector
+ if vector.HasField("dense"):
+ return Vectors.dense(vector.dense.value)
+ raise ValueError("TODO, support sparse vector")
+
+ if param.HasField("matrix"):
+ matrix = param.matrix
+ # TODO support sparse matrix
+ if matrix.HasField("dense") and not matrix.dense.is_transposed:
+ return Matrices.dense(
+ matrix.dense.num_rows,
+ matrix.dense.num_cols,
+ matrix.dense.value,
+ )
+ raise ValueError("TODO, support sparse matrix")
Review Comment:
Hmm, I will support it in this PR.
##########
python/pyspark/ml/tests/test_training_summary.py:
##########
@@ -122,94 +119,6 @@ def test_glr_summary(self):
sameSummary = model.evaluate(df)
self.assertAlmostEqual(sameSummary.deviance, s.deviance)
- def test_binary_logistic_regression_summary(self):
Review Comment:
I've moved the tests into test_classification.py which is the base class for both classic spark and connect spark. See https://github.com/apache/spark/pull/48791/files#diff-9bb8f507634f33717f8758848a6369d9f57a0aeb6629b16648a38fbd15352892R39 and https://github.com/apache/spark/pull/48791/files#diff-a4da55d9f9f7dd55641f346cffe7ee87432f5ca06da40a75dddca514e6c6a648R184
##########
python/pyspark/ml/remote/util.py:
##########
@@ -0,0 +1,259 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import functools
+import os
+from typing import Any, cast, TypeVar, Callable, TYPE_CHECKING, Type
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, serialize, deserialize
+from pyspark.sql import is_remote
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+
+if TYPE_CHECKING:
+ from pyspark.ml.wrapper import JavaWrapper, JavaEstimator
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+
+FuncT = TypeVar("FuncT", bound=Callable[..., Any])
+
+
+def try_remote_intermediate_result(f: FuncT) -> FuncT:
+ """Mark the function/property that returns the intermediate result of the remote call.
+ Eg, model.summary"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper") -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ return f"{self._java_obj}.{f.__name__}"
+ else:
+ return f(self)
+
+ return cast(FuncT, wrapped)
+
+
+def try_remote_attribute_relation(f: FuncT) -> FuncT:
+ """Mark the function/property that returns a Relation.
+ Eg, model.summary.roc"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper", *args: Any, **kwargs: Any) -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ # The attribute returns a dataframe, we need to wrap it
+ # in the _ModelAttributeRelationPlan
+ from pyspark.ml.remote.proto import _ModelAttributeRelationPlan
+ from pyspark.sql.connect.session import SparkSession
+
+ assert isinstance(self._java_obj, str)
Review Comment:
This assert here is typically for the python typing issue. We're reusing`self._java_obj` in the PR, but the self._java_obj is a "JavaObject" in the classic spark.
##########
python/pyspark/ml/remote/serialize.py:
##########
@@ -0,0 +1,120 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from typing import Any, List, TYPE_CHECKING, Mapping, Optional
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.linalg import (
+ Vectors,
+ Matrices,
+ DenseVector,
+ SparseVector,
+ DenseMatrix,
+ SparseMatrix,
+)
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+from pyspark.sql.connect.expressions import LiteralExpression
+
+if TYPE_CHECKING:
+ from pyspark.sql.connect.client import SparkConnectClient
+ from pyspark.ml.param import Params
+
+
+def serialize_param(value: Any, client: "SparkConnectClient") -> pb2.Param:
+ if isinstance(value, DenseVector):
+ return pb2.Param(vector=pb2.Vector(dense=pb2.Vector.Dense(value=value.values.tolist())))
+ elif isinstance(value, SparseVector):
+ return pb2.Param(
+ vector=pb2.Vector(
+ sparse=pb2.Vector.Sparse(
+ size=value.size, index=value.indices.tolist(), value=value.values.tolist()
+ )
+ )
+ )
+ elif isinstance(value, DenseMatrix):
+ return pb2.Param(
+ matrix=pb2.Matrix(
+ dense=pb2.Matrix.Dense(
+ num_rows=value.numRows, num_cols=value.numCols, value=value.values.tolist()
+ )
+ )
+ )
+ elif isinstance(value, SparseMatrix):
+ return pb2.Param(
+ matrix=pb2.Matrix(
+ sparse=pb2.Matrix.Sparse(
+ num_rows=value.numRows,
+ num_cols=value.numCols,
+ colptr=value.colPtrs.tolist(),
+ row_index=value.rowIndices.tolist(),
+ value=value.values.tolist(),
+ )
+ )
+ )
+ else:
+ literal = LiteralExpression._from_value(value).to_plan(client).literal
+ return pb2.Param(literal=literal)
+
+
+def serialize(client: "SparkConnectClient", *args: Any) -> List[Any]:
+ result = []
+ for arg in args:
+ if isinstance(arg, RemoteDataFrame):
+ result.append(pb2.FetchModelAttr.Args(input=arg._plan.plan(client)))
+ else:
+ result.append(pb2.FetchModelAttr.Args(param=serialize_param(arg, client)))
+ return result
+
+
+def deserialize_param(param: pb2.Param) -> Any:
+ if param.HasField("literal"):
+ return LiteralExpression._to_value(param.literal)
+ if param.HasField("vector"):
+ vector = param.vector
+ # TODO support sparse vector
+ if vector.HasField("dense"):
+ return Vectors.dense(vector.dense.value)
+ raise ValueError("TODO, support sparse vector")
+
+ if param.HasField("matrix"):
+ matrix = param.matrix
+ # TODO support sparse matrix
+ if matrix.HasField("dense") and not matrix.dense.is_transposed:
+ return Matrices.dense(
+ matrix.dense.num_rows,
+ matrix.dense.num_cols,
+ matrix.dense.value,
+ )
+ raise ValueError("TODO, support sparse matrix")
+
+ raise ValueError("Unsupported param type")
+
+
+def deserialize(ml_command_result: Optional[pb2.MlCommandResponse]) -> Any:
+ assert ml_command_result is not None
+ if ml_command_result.HasField("operator_info"):
+ return ml_command_result.operator_info
+
+ if ml_command_result.HasField("param"):
+ return deserialize_param(ml_command_result.param)
+ raise ValueError()
Review Comment:
Sounds good. Done
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -1143,6 +1148,26 @@ def execute_command_as_iterator(
},
)
+ def execute_ml(self, req: pb2.ExecutePlanRequest) -> Optional[pb2.MlCommandResponse]:
+ """
+ Execute the ML command request and return ML response result
+ Parameters
+ ----------
+ req : pb2.ExecutePlanRequest
+ Proto representation of the plan.
+ """
+ logger.info("Execute ML")
+ try:
+ for attempt in self._retrying():
+ with attempt:
+ for b in self._stub.ExecutePlan(req, metadata=self._builder.metadata()):
+ if b.HasField("ml_command_result"):
+ return b.ml_command_result
Review Comment:
Yeah. the new commit have moved the ml_command to the existing command. That's really cool.
##########
python/pyspark/ml/remote/util.py:
##########
@@ -0,0 +1,259 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import functools
+import os
+from typing import Any, cast, TypeVar, Callable, TYPE_CHECKING, Type
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, serialize, deserialize
+from pyspark.sql import is_remote
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+
+if TYPE_CHECKING:
+ from pyspark.ml.wrapper import JavaWrapper, JavaEstimator
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+
+FuncT = TypeVar("FuncT", bound=Callable[..., Any])
+
+
+def try_remote_intermediate_result(f: FuncT) -> FuncT:
+ """Mark the function/property that returns the intermediate result of the remote call.
+ Eg, model.summary"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper") -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ return f"{self._java_obj}.{f.__name__}"
+ else:
+ return f(self)
+
+ return cast(FuncT, wrapped)
+
+
+def try_remote_attribute_relation(f: FuncT) -> FuncT:
+ """Mark the function/property that returns a Relation.
+ Eg, model.summary.roc"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper", *args: Any, **kwargs: Any) -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ # The attribute returns a dataframe, we need to wrap it
+ # in the _ModelAttributeRelationPlan
+ from pyspark.ml.remote.proto import _ModelAttributeRelationPlan
+ from pyspark.sql.connect.session import SparkSession
+
+ assert isinstance(self._java_obj, str)
+ plan = _ModelAttributeRelationPlan(self._java_obj, f.__name__)
+ session = SparkSession.getActiveSession()
+ assert session is not None
+ return RemoteDataFrame(plan, session)
+ else:
+ return f(self, *args, **kwargs)
+
+ return cast(FuncT, wrapped)
+
+
+def try_remote_fit(f: FuncT) -> FuncT:
+ """Mark the function that fits a model."""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaEstimator", dataset: RemoteDataFrame) -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ client = dataset.sparkSession.client
+ input = dataset._plan.plan(client)
+ assert isinstance(self._java_obj, str)
Review Comment:
Yeah, mainly for the typing issue.
##########
python/pyspark/ml/tests/test_persistence.py:
##########
@@ -153,29 +153,6 @@ def test_linear_regression_pmml_basic(self):
self.assertIn("Apache Spark", pmml_text)
self.assertIn("PMML", pmml_text)
- def test_logistic_regression(self):
Review Comment:
It's in test_classification.py. https://github.com/apache/spark/pull/48791/files#diff-a4da55d9f9f7dd55641f346cffe7ee87432f5ca06da40a75dddca514e6c6a648R236
##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -38,6 +39,7 @@ message Plan {
oneof op_type {
Relation root = 1;
Command command = 2;
+ MlCommand ml_command = 3;
}
}
Review Comment:
Really appreciate the valuable suggestions. Done.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteThreadRunner.scala:
##########
@@ -311,6 +313,17 @@ private[connect] class ExecuteThreadRunner(executeHolder: ExecuteHolder) extends
planner.process(command = command, responseObserver = responseObserver)
}
+ private def handleMLCommand(request: proto.ExecutePlanRequest): Unit = {
Review Comment:
Yes, really appreciate the comments. after putting the ML commands into existing commands, life is becoming easier.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLCache.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.ml
+
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.Model
+
+// TODO need to support persistence for model if memory is tight
Review Comment:
Sounds good, will have one.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.Model
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.MLUtils.loadModel
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+private class ModelAttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ private lazy val model = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(model.asInstanceOf[Object]) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(model.asInstanceOf[Object], lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(model.asInstanceOf[Object]) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchModelAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+object MLHandler extends Logging {
Review Comment:
Added some documentation like "MLHandler is a utility to group all ML operations".
> Why can it be an object
Not have a strong reason it must be an object. Any idea about it?
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Model, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw new RuntimeException("Unsupported parameter type")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the fit command
+ *
+ * @param fit
+ * command
+ * @return
+ * an Estimator
+ */
+ def getEstimator(fit: proto.MlCommand.Fit): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the fit command
+ val name = fit.getEstimator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw new RuntimeException(s"Failed to find estimator: $name")
+ }
+ val uid = fit.getEstimator.getUid
+ val estimator: Estimator[_] = estimators(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Estimator[_]]
+
+ // Set parameters for the estimator
+ val params = fit.getParams
+ MLUtils.setInstanceParams(estimator, params)
+ estimator
+ }
+
+ def loadModel(className: String, path: String): Model[_] = {
+ // scalastyle:off classforname
+ val clazz = Class.forName(className)
+ // scalastyle:on classforname
+ val model = invokeStaticMethod(clazz, "load", path)
+ model.asInstanceOf[Model[_]]
+ }
+
+ /**
+ * Get the transformer instance according to the transform proto
+ *
+ * @param transformProto
+ * transform proto
+ * @return
+ * a Transformer
+ */
+ def getTransformer(transformProto: proto.MlRelation.Transform): Transformer = {
+ // Get the transformer name
+ val name = transformProto.getTransformer.getName
+ if (transformers.isEmpty || !transformers.contains(name)) {
+ throw new RuntimeException(s"Failed to find transformer: $name")
+ }
+ val uid = transformProto.getTransformer.getUid
+ val transformer = transformers(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Transformer]
+
+ val params = transformProto.getParams
+ MLUtils.setInstanceParams(transformer, params)
+ transformer
+ }
+
+ private lazy val ALLOWED_ATTRIBUTES = HashSet(
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1908683079
##########
sql/connect/common/src/main/protobuf/spark/connect/ml_common.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// MlParams stores param settings for ML Estimator / Transformer / Evaluator
+message MlParams {
+ // User-supplied params
+ map<string, Param> params = 1;
+}
+
+// Represents the parameter type of the ML instance, or the returned value
+// of the attribute
+message Param {
+ oneof param_type {
+ Expression.Literal literal = 1;
+ Vector vector = 2;
Review Comment:
I also think we can optimize out the `message Param`:
option 1: support `Vector` and `Matrix` in `message Literal`;
option 2: Using `message Expression` or `Expression.Literal.Struct` to express the parameters;
##########
mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Estimator:
##########
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Spark Connect ML uses ServiceLoader to find out the supported Spark Ml estimators.
+# So register the supported estimator here if you're trying to add a new one.
+org.apache.spark.ml.classification.LogisticRegression
Review Comment:
```suggestion
org.apache.spark.ml.classification.LogisticRegression
```
##########
mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Transformer:
##########
@@ -0,0 +1,20 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Spark Connect ML uses ServiceLoader to find out the supported Spark Ml non-model transformer.
+# So register the supported transformer here if you're trying to add a new one.
+org.apache.spark.ml.feature.VectorAssembler
Review Comment:
```suggestion
org.apache.spark.ml.feature.VectorAssembler
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1856354610
##########
python/pyspark/ml/remote/readwrite.py:
##########
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import cast, Type, TYPE_CHECKING
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, deserialize, deserialize_param
+from pyspark.ml.util import MLWriter, MLReader, RL
+from pyspark.ml.wrapper import JavaWrapper
+
+if TYPE_CHECKING:
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+ from pyspark.core.context import SparkContext
+
+
+class RemoteMLWriter(MLWriter):
+ def __init__(self, instance: "JavaMLWritable") -> None:
+ super().__init__()
+ self._instance = instance
+
+ @property
+ def sc(self) -> "SparkContext":
+ raise RuntimeError("Accessing SparkContext is not supported on Connect")
+
+ def save(self, path: str) -> None:
+ from pyspark.ml.wrapper import JavaModel
+
+ if isinstance(self._instance, JavaModel):
+ from pyspark.sql.connect.session import SparkSession
+
+ session = SparkSession.getActiveSession()
Review Comment:
Hmm, I couldn't find a proper way to pass the session. Any suggestions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1908683079
##########
sql/connect/common/src/main/protobuf/spark/connect/ml_common.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// MlParams stores param settings for ML Estimator / Transformer / Evaluator
+message MlParams {
+ // User-supplied params
+ map<string, Param> params = 1;
+}
+
+// Represents the parameter type of the ML instance, or the returned value
+// of the attribute
+message Param {
+ oneof param_type {
+ Expression.Literal literal = 1;
+ Vector vector = 2;
Review Comment:
I also think we can optimize out the `message Param`:
option 1: support `Vector` and `Matrix` in `message Literal`;
option 2 (Preferred): Using `message Expression` or `Expression.Literal.Struct` to express the parameters;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2581450570
Hi @zhengruifeng,
> I also think we can optimize out the message Param:
option 1: support Vector and Matrix in message Literal;
option 2 (Preferred): Using message Expression or Expression.Literal.Struct to express the parameters;
If you insist on optimizing it, I would like to add vector/matrix in the Literal. What's your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1915951762
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
Review Comment:
Is this used, @wbo4958?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2538028107
> Hello. I suppose many design of this pr, especially the proto and server part, are similar to that in #40479. So maybe at least this pr#40479 should at least be mentioned?
Yeah, If you have looked at the [JIRA here](https://issues.apache.org/jira/browse/SPARK-49907) and there is a [design doc](https://docs.google.com/document/d/1EUvSZuI-so83cxb_fTVMoz0vUfAaFmqXt39yoHI-D9I/edit?tab=t.0#heading=h.669989ni9rv0) where you can see @WeichenXu123 PR.
But you're right, I should put the original PR in the description of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876694332
##########
mllib/src/main/scala/org/apache/spark/ml/util/Summary.scala:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.util
+
+import org.apache.spark.annotation.Since
+
+/**
+ * Trait for the Summary
+ */
+@Since("4.0.0")
+private[spark] trait Summary
Review Comment:
Why do we need this trait? The documentation could use some work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876959198
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -97,13 +98,60 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
+ oneof ml_type {
+ Transform transform = 1;
+ FetchAttr fetch_attr = 2;
+ }
+ // Relation to represent transform(input) of the operator
+ // which could be a cached model or a new transformer
+ message Transform {
+ oneof operator {
+ // Object reference
+ ObjectRef obj_ref = 1;
+ // Could be an ML transformer like VectorAssembler
+ MlOperator transformer = 2;
+ }
+ // the input dataframe
+ Relation input = 3;
+ // the operator specific parameters
+ MlParams params = 4;
+ }
+}
+
+// Message for fetching attribute from object on the server side.
+// FetchAttr can be represented as a Relation or a ML command
+// Eg, model.coefficients, model.summary.weightedPrecision
+// or model.summary.roc which returns a DataFrame
+message FetchAttr {
+ // (Required) reference to the object on the server side or
+ // the intermediate attribute of the model. eg, "model.summary"
+ ObjectRef obj_ref = 1;
+ // (Required) the method name. Eg, "coefficients" of the model
+ // and "weightedPrecision" of "model.summary"
+ string method = 2;
+ // (Optional) the parameters of the method
+ repeated Args args = 3;
+
+ message Args {
Review Comment:
Should this support named arguments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876957116
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
+ oneof command {
Review Comment:
It would be good to explain what these commands actually do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1885825650
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
Review Comment:
please add a
```
option go_package = "internal/generated";
```
##########
sql/connect/common/src/main/protobuf/spark/connect/ml_common.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
Review Comment:
please add a
```
option go_package = "internal/generated";
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1883061017
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
Review Comment:
Huh, I see. Seems it's doable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1886000855
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
Review Comment:
I'm wondering if I should move the commands and result-of-commands from ml.proto to command.proto and base.proto respectively, so after that, there will be only ml_common.proto left.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng closed pull request #48791: [SPARK-49907][ML][CONNECT] Support spark.ml on Connect
URL: https://github.com/apache/spark/pull/48791
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876941962
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
Review Comment:
A small follow-up question. We are using fully qualified names for resolution. In that case you don't really need the service loader because you can just use `SparkClassUtils.classForName(..)`. Are there plans to deviate from this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876992507
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ protected lazy val instance = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(instance, lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+}
+
+private class ModelAttributeHelper(
+ sessionHolder: SessionHolder,
+ objIdentifier: String,
+ method: Option[String],
+ argValues: Array[Object] = Array.empty,
+ argClasses: Array[Class[_]] = Array.empty)
+ extends AttributeHelper(sessionHolder, objIdentifier, method, argValues, argClasses) {
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val model = instance.asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object AttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): AttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new AttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+// MLHandler is a utility to group all ML operations
+object MLHandler extends Logging {
+ def handleMlCommand(
+ sessionHolder: SessionHolder,
+ mlCommand: proto.MlCommand): proto.MlCommandResult = {
+
+ val mlCache = sessionHolder.mlCache
+
+ mlCommand.getCommandCase match {
+ case proto.MlCommand.CommandCase.FIT =>
+ val fitCmd = mlCommand.getFit
+ val estimatorProto = fitCmd.getEstimator
+ assert(estimatorProto.getType == proto.MlOperator.OperatorType.ESTIMATOR)
+
+ val dataset = MLUtils.parseRelationProto(fitCmd.getDataset, sessionHolder)
+ val estimator = MLUtils.getEstimator(estimatorProto, Some(fitCmd.getParams))
+ val model = estimator.fit(dataset).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id)))
+ .build()
+
+ case proto.MlCommand.CommandCase.FETCH_ATTR =>
+ val args = mlCommand.getFetchAttr.getArgsList.asScala.toArray
+ val helper = AttributeHelper(
+ sessionHolder,
+ mlCommand.getFetchAttr.getObjRef.getId,
+ Option(mlCommand.getFetchAttr.getMethod),
+ args)
+ val attrResult = helper.getAttribute
+ attrResult match {
+ case s: Summary =>
+ val id = mlCache.register(s)
+ proto.MlCommandResult.newBuilder().setSummary(id).build()
+ case _ =>
+ val param = Serializer.serializeParam(attrResult)
+ proto.MlCommandResult.newBuilder().setParam(param).build()
+ }
+
+ case proto.MlCommand.CommandCase.DELETE =>
+ val modelId = mlCommand.getDelete.getObjRef.getId
+ var result = false
+ if (!modelId.contains(".")) {
+ mlCache.remove(modelId)
+ result = true
+ }
+ proto.MlCommandResult
+ .newBuilder()
+ .setParam(
+ proto.Param
+ .newBuilder()
+ .setLiteral(LiteralValueProtoConverter.toLiteralProto(result))
+ .build())
+ .build()
+
+ case proto.MlCommand.CommandCase.WRITE =>
+ mlCommand.getWrite.getTypeCase match {
+ case proto.MlCommand.Writer.TypeCase.OBJ_REF => // save a model
+ val modelId = mlCommand.getWrite.getObjRef.getId
+ val model = mlCache.get(modelId).asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, mlCommand.getWrite.getParams)
+
+ copiedModel match {
+ case m: MLWritable => MLUtils.write(m, mlCommand.getWrite)
+ case other => throw MlUnsupportedException(s"$other is not writable")
+ }
+
+ // save an estimator/evaluator/transformer
+ case proto.MlCommand.Writer.TypeCase.OPERATOR =>
+ val writer = mlCommand.getWrite
+ if (writer.getOperator.getType == proto.MlOperator.OperatorType.ESTIMATOR) {
+ val estimator = MLUtils.getEstimator(writer.getOperator, Some(writer.getParams))
+ estimator match {
+ case m: MLWritable => MLUtils.write(m, mlCommand.getWrite)
+ case other => throw MlUnsupportedException(s"Estimator $other is not writable")
+ }
+ } else {
+ throw MlUnsupportedException(s"${writer.getOperator.getName} not supported")
+ }
+
+ case other => throw MlUnsupportedException(s"$other not supported")
+ }
+ proto.MlCommandResult.newBuilder().build()
+
+ case proto.MlCommand.CommandCase.READ =>
+ val operator = mlCommand.getRead.getOperator
+ val name = operator.getName
+ val path = mlCommand.getRead.getPath
+
+ if (operator.getType == proto.MlOperator.OperatorType.MODEL) {
+ val model = MLUtils.load(name, path).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id))
+ .setUid(model.uid)
+ .setParams(Serializer.serializeParams(model)))
+ .build()
+
+ } else if (operator.getType == proto.MlOperator.OperatorType.ESTIMATOR) {
+ val estimator = MLUtils.load(name, path).asInstanceOf[Estimator[_]]
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setName(name)
+ .setUid(estimator.uid)
+ .setParams(Serializer.serializeParams(estimator)))
+ .build()
+ } else {
+ throw MlUnsupportedException(s"${operator.getType} not supported")
+ }
+
+ case other => throw MlUnsupportedException(s"$other not supported")
+ }
+ }
+
+ def transformMLRelation(relation: proto.MlRelation, sessionHolder: SessionHolder): DataFrame = {
Review Comment:
This should return a LogicalPlan...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876984770
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
+ oneof command {
+ Fit fit = 1;
+ FetchAttr fetch_attr = 2;
+ Delete delete = 3;
+ Writer write = 4;
Review Comment:
Use `Write` instead of `Writer`? You are using verbs for all the other commands. A verb IMO instead of a noun is more fitting for a command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2500773213
Hi @grundprinzip, Thx for your trying out. I even haven't tested other estimators except LogistcRegression and transformers except VectorAssembler. Definitely, the *Evaluator* has not been supported yet, but I think it's just a trivial work to make it.
For the feature engineering algorithms, just like I said, I haven't tested them.
I originally planned to support other estimatores/transformers (which should be testing work, I guess)in the followup PR. So Do you want me to support all the ML things in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1849905072
##########
mllib/src/main/resources/META-INF/services/org.apache.spark.ml.Estimator:
##########
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.spark.ml.classification.LogisticRegression
Review Comment:
Should we leave a comment here about how to add new Estimators?
##########
mllib/src/main/scala/org/apache/spark/ml/param/params.scala:
##########
@@ -45,9 +45,14 @@ import org.apache.spark.util.ArrayImplicits._
* See [[ParamValidators]] for factory methods for common validation functions.
* @tparam T param value type
*/
-class Param[T](val parent: String, val name: String, val doc: String, val isValid: T => Boolean)
+class Param[T: ClassTag](
+ val parent: String, val name: String, val doc: String, val isValid: T => Boolean)
extends Serializable {
+ // spark connect ML needs T type information which has been erased when compiling,
Review Comment:
```suggestion
// Spark Connect ML needs T type information which has been erased when compiling,
```
##########
python/pyspark/ml/remote/readwrite.py:
##########
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import cast, Type, TYPE_CHECKING
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, deserialize, deserialize_param
+from pyspark.ml.util import MLWriter, MLReader, RL
+from pyspark.ml.wrapper import JavaWrapper
+
+if TYPE_CHECKING:
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+ from pyspark.core.context import SparkContext
+
+
+class RemoteMLWriter(MLWriter):
+ def __init__(self, instance: "JavaMLWritable") -> None:
+ super().__init__()
+ self._instance = instance
+
+ @property
+ def sc(self) -> "SparkContext":
+ raise RuntimeError("Accessing SparkContext is not supported on Connect")
+
+ def save(self, path: str) -> None:
+ from pyspark.ml.wrapper import JavaModel
+
+ if isinstance(self._instance, JavaModel):
+ from pyspark.sql.connect.session import SparkSession
+
+ session = SparkSession.getActiveSession()
+ assert session is not None
+ instance = cast("JavaModel", self._instance)
+ params = serialize_ml_params(instance, session.client)
+
+ assert isinstance(instance._java_obj, str)
+ writer = pb2.MlCommand.Writer(
+ model_ref=pb2.ModelRef(id=instance._java_obj),
+ params=params,
+ path=path,
+ should_overwrite=self.shouldOverwrite,
+ options=self.optionMap,
+ )
+ req = session.client._execute_plan_request_with_metadata()
+ req.plan.ml_command.write.CopyFrom(writer)
+ session.client.execute_ml(req)
+
+
+class RemoteMLReader(MLReader[RL]):
+ def __init__(self, clazz: Type["JavaMLReadable[RL]"]) -> None:
+ super().__init__()
+ self._clazz = clazz
Review Comment:
Is this generalizable because of the consistencies of the type names in general?
##########
python/pyspark/ml/remote/util.py:
##########
@@ -0,0 +1,259 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import functools
+import os
+from typing import Any, cast, TypeVar, Callable, TYPE_CHECKING, Type
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, serialize, deserialize
+from pyspark.sql import is_remote
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+
+if TYPE_CHECKING:
+ from pyspark.ml.wrapper import JavaWrapper, JavaEstimator
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+
+FuncT = TypeVar("FuncT", bound=Callable[..., Any])
+
+
+def try_remote_intermediate_result(f: FuncT) -> FuncT:
+ """Mark the function/property that returns the intermediate result of the remote call.
+ Eg, model.summary"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper") -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ return f"{self._java_obj}.{f.__name__}"
+ else:
+ return f(self)
+
+ return cast(FuncT, wrapped)
+
+
+def try_remote_attribute_relation(f: FuncT) -> FuncT:
+ """Mark the function/property that returns a Relation.
+ Eg, model.summary.roc"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper", *args: Any, **kwargs: Any) -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ # The attribute returns a dataframe, we need to wrap it
+ # in the _ModelAttributeRelationPlan
+ from pyspark.ml.remote.proto import _ModelAttributeRelationPlan
+ from pyspark.sql.connect.session import SparkSession
+
+ assert isinstance(self._java_obj, str)
+ plan = _ModelAttributeRelationPlan(self._java_obj, f.__name__)
+ session = SparkSession.getActiveSession()
+ assert session is not None
+ return RemoteDataFrame(plan, session)
+ else:
+ return f(self, *args, **kwargs)
+
+ return cast(FuncT, wrapped)
+
+
+def try_remote_fit(f: FuncT) -> FuncT:
+ """Mark the function that fits a model."""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaEstimator", dataset: RemoteDataFrame) -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ client = dataset.sparkSession.client
+ input = dataset._plan.plan(client)
+ assert isinstance(self._java_obj, str)
Review Comment:
similar here
##########
python/pyspark/ml/tests/test_training_summary.py:
##########
@@ -122,94 +119,6 @@ def test_glr_summary(self):
sameSummary = model.evaluate(df)
self.assertAlmostEqual(sameSummary.deviance, s.deviance)
- def test_binary_logistic_regression_summary(self):
Review Comment:
where did these tests move to?
##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -38,6 +39,7 @@ message Plan {
oneof op_type {
Relation root = 1;
Command command = 2;
+ MlCommand ml_command = 3;
}
}
Review Comment:
Can we model the ML Operation as a command directly instead?
##########
python/pyspark/ml/remote/readwrite.py:
##########
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import cast, Type, TYPE_CHECKING
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, deserialize, deserialize_param
+from pyspark.ml.util import MLWriter, MLReader, RL
+from pyspark.ml.wrapper import JavaWrapper
+
+if TYPE_CHECKING:
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+ from pyspark.core.context import SparkContext
+
+
+class RemoteMLWriter(MLWriter):
+ def __init__(self, instance: "JavaMLWritable") -> None:
+ super().__init__()
+ self._instance = instance
+
+ @property
+ def sc(self) -> "SparkContext":
+ raise RuntimeError("Accessing SparkContext is not supported on Connect")
+
+ def save(self, path: str) -> None:
+ from pyspark.ml.wrapper import JavaModel
+
+ if isinstance(self._instance, JavaModel):
+ from pyspark.sql.connect.session import SparkSession
+
+ session = SparkSession.getActiveSession()
+ assert session is not None
+ instance = cast("JavaModel", self._instance)
+ params = serialize_ml_params(instance, session.client)
+
+ assert isinstance(instance._java_obj, str)
+ writer = pb2.MlCommand.Writer(
+ model_ref=pb2.ModelRef(id=instance._java_obj),
+ params=params,
+ path=path,
+ should_overwrite=self.shouldOverwrite,
+ options=self.optionMap,
+ )
+ req = session.client._execute_plan_request_with_metadata()
+ req.plan.ml_command.write.CopyFrom(writer)
+ session.client.execute_ml(req)
+
+
+class RemoteMLReader(MLReader[RL]):
+ def __init__(self, clazz: Type["JavaMLReadable[RL]"]) -> None:
+ super().__init__()
+ self._clazz = clazz
+
+ def load(self, path: str) -> RL:
+ from pyspark.sql.connect.session import SparkSession
+
+ session = SparkSession.getActiveSession()
+ assert session is not None
+
+ java_package = (
Review Comment:
This probably deserves a comment here.
##########
python/pyspark/ml/remote/readwrite.py:
##########
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import cast, Type, TYPE_CHECKING
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, deserialize, deserialize_param
+from pyspark.ml.util import MLWriter, MLReader, RL
+from pyspark.ml.wrapper import JavaWrapper
+
+if TYPE_CHECKING:
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+ from pyspark.core.context import SparkContext
+
+
+class RemoteMLWriter(MLWriter):
+ def __init__(self, instance: "JavaMLWritable") -> None:
+ super().__init__()
+ self._instance = instance
+
+ @property
+ def sc(self) -> "SparkContext":
+ raise RuntimeError("Accessing SparkContext is not supported on Connect")
+
+ def save(self, path: str) -> None:
+ from pyspark.ml.wrapper import JavaModel
+
+ if isinstance(self._instance, JavaModel):
Review Comment:
Somewhere we need to put an explanation that gives a more detailed understanding why a JavaModel instance can work with Spark Connect that doesn't have any JVM locally.
##########
python/pyspark/ml/remote/serialize.py:
##########
@@ -0,0 +1,120 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from typing import Any, List, TYPE_CHECKING, Mapping, Optional
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.linalg import (
+ Vectors,
+ Matrices,
+ DenseVector,
+ SparseVector,
+ DenseMatrix,
+ SparseMatrix,
+)
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+from pyspark.sql.connect.expressions import LiteralExpression
+
+if TYPE_CHECKING:
+ from pyspark.sql.connect.client import SparkConnectClient
+ from pyspark.ml.param import Params
+
+
+def serialize_param(value: Any, client: "SparkConnectClient") -> pb2.Param:
+ if isinstance(value, DenseVector):
+ return pb2.Param(vector=pb2.Vector(dense=pb2.Vector.Dense(value=value.values.tolist())))
+ elif isinstance(value, SparseVector):
+ return pb2.Param(
+ vector=pb2.Vector(
+ sparse=pb2.Vector.Sparse(
+ size=value.size, index=value.indices.tolist(), value=value.values.tolist()
+ )
+ )
+ )
+ elif isinstance(value, DenseMatrix):
+ return pb2.Param(
+ matrix=pb2.Matrix(
+ dense=pb2.Matrix.Dense(
+ num_rows=value.numRows, num_cols=value.numCols, value=value.values.tolist()
+ )
+ )
+ )
+ elif isinstance(value, SparseMatrix):
+ return pb2.Param(
+ matrix=pb2.Matrix(
+ sparse=pb2.Matrix.Sparse(
+ num_rows=value.numRows,
+ num_cols=value.numCols,
+ colptr=value.colPtrs.tolist(),
+ row_index=value.rowIndices.tolist(),
+ value=value.values.tolist(),
+ )
+ )
+ )
+ else:
+ literal = LiteralExpression._from_value(value).to_plan(client).literal
+ return pb2.Param(literal=literal)
+
+
+def serialize(client: "SparkConnectClient", *args: Any) -> List[Any]:
+ result = []
+ for arg in args:
+ if isinstance(arg, RemoteDataFrame):
+ result.append(pb2.FetchModelAttr.Args(input=arg._plan.plan(client)))
+ else:
+ result.append(pb2.FetchModelAttr.Args(param=serialize_param(arg, client)))
+ return result
+
+
+def deserialize_param(param: pb2.Param) -> Any:
+ if param.HasField("literal"):
+ return LiteralExpression._to_value(param.literal)
+ if param.HasField("vector"):
+ vector = param.vector
+ # TODO support sparse vector
+ if vector.HasField("dense"):
+ return Vectors.dense(vector.dense.value)
+ raise ValueError("TODO, support sparse vector")
+
+ if param.HasField("matrix"):
+ matrix = param.matrix
+ # TODO support sparse matrix
+ if matrix.HasField("dense") and not matrix.dense.is_transposed:
+ return Matrices.dense(
+ matrix.dense.num_rows,
+ matrix.dense.num_cols,
+ matrix.dense.value,
+ )
+ raise ValueError("TODO, support sparse matrix")
Review Comment:
Should we track this with a child JIRA?
##########
python/pyspark/ml/remote/readwrite.py:
##########
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import cast, Type, TYPE_CHECKING
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, deserialize, deserialize_param
+from pyspark.ml.util import MLWriter, MLReader, RL
+from pyspark.ml.wrapper import JavaWrapper
+
+if TYPE_CHECKING:
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+ from pyspark.core.context import SparkContext
+
+
+class RemoteMLWriter(MLWriter):
+ def __init__(self, instance: "JavaMLWritable") -> None:
+ super().__init__()
+ self._instance = instance
+
+ @property
+ def sc(self) -> "SparkContext":
+ raise RuntimeError("Accessing SparkContext is not supported on Connect")
+
+ def save(self, path: str) -> None:
+ from pyspark.ml.wrapper import JavaModel
+
+ if isinstance(self._instance, JavaModel):
+ from pyspark.sql.connect.session import SparkSession
+
+ session = SparkSession.getActiveSession()
+ assert session is not None
+ instance = cast("JavaModel", self._instance)
+ params = serialize_ml_params(instance, session.client)
+
+ assert isinstance(instance._java_obj, str)
+ writer = pb2.MlCommand.Writer(
+ model_ref=pb2.ModelRef(id=instance._java_obj),
+ params=params,
+ path=path,
+ should_overwrite=self.shouldOverwrite,
+ options=self.optionMap,
+ )
+ req = session.client._execute_plan_request_with_metadata()
+ req.plan.ml_command.write.CopyFrom(writer)
+ session.client.execute_ml(req)
+
+
+class RemoteMLReader(MLReader[RL]):
+ def __init__(self, clazz: Type["JavaMLReadable[RL]"]) -> None:
+ super().__init__()
+ self._clazz = clazz
+
+ def load(self, path: str) -> RL:
+ from pyspark.sql.connect.session import SparkSession
+
+ session = SparkSession.getActiveSession()
+ assert session is not None
+
+ java_package = (
+ self._clazz.__module__.replace("pyspark", "org.apache.spark")
+ + "."
+ + self._clazz.__name__
+ )
+ reader = pb2.MlCommand.Reader(clazz=java_package, path=path)
+ req = session.client._execute_plan_request_with_metadata()
+ req.plan.ml_command.read.CopyFrom(reader)
+ model_info = deserialize(session.client.execute_ml(req))
+ session.client.add_ml_model(model_info.model_ref.id)
+
+ # bypass the typing
+ def _get_class() -> Type[RL]:
+ parts = (self._clazz.__module__ + "." + self._clazz.__name__).split(".")
+ module = ".".join(parts[:-1])
+ m = __import__(module, fromlist=[parts[-1]])
+ return getattr(m, parts[-1])
+
+ py_type = _get_class()
+ if issubclass(py_type, JavaWrapper):
+ instance = py_type(model_info.model_ref.id)
+ instance._resetUid(model_info.uid)
+ params = {k: deserialize_param(v) for k, v in model_info.params.params.items()}
+ instance._set(**params)
+ return instance
+ else:
Review Comment:
Would be great to get more explanation for this one here.
##########
python/pyspark/ml/remote/readwrite.py:
##########
@@ -0,0 +1,100 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from typing import cast, Type, TYPE_CHECKING
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, deserialize, deserialize_param
+from pyspark.ml.util import MLWriter, MLReader, RL
+from pyspark.ml.wrapper import JavaWrapper
+
+if TYPE_CHECKING:
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+ from pyspark.core.context import SparkContext
+
+
+class RemoteMLWriter(MLWriter):
+ def __init__(self, instance: "JavaMLWritable") -> None:
+ super().__init__()
+ self._instance = instance
+
+ @property
+ def sc(self) -> "SparkContext":
+ raise RuntimeError("Accessing SparkContext is not supported on Connect")
+
+ def save(self, path: str) -> None:
+ from pyspark.ml.wrapper import JavaModel
+
+ if isinstance(self._instance, JavaModel):
+ from pyspark.sql.connect.session import SparkSession
+
+ session = SparkSession.getActiveSession()
Review Comment:
Is there a different way we can get to the session than relying on the active session here? Previously we've tried to pass the session through to avoid weird confusions.
##########
python/pyspark/ml/remote/proto.py:
##########
@@ -0,0 +1,86 @@
+#
Review Comment:
We're a bit in a bad position here because, high-level we should use the `pyspark.ml.connect` module, but this is occupied by the previous approach of the implementation.
I'm wondering if we should do some cleanup later to delete the previous code and move this one to the right package.
@WeichenXu123 what do you think?
##########
python/pyspark/ml/tests/connect/test_connect_spark_ml_classification.py:
##########
@@ -0,0 +1,59 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import os
+import unittest
+
+from pyspark.ml.tests.test_classification import ClassificationTestsMixin
+from pyspark.sql import SparkSession
+
+
+class ClassificationTestsOnConnect(ClassificationTestsMixin, unittest.TestCase):
Review Comment:
@HyukjinKwon do you mind having a look at the pattern here, it looks sound, but I want to make sure it's similar to what we do already with Spark Connect parity tests.
##########
python/pyspark/ml/remote/serialize.py:
##########
@@ -0,0 +1,120 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+from typing import Any, List, TYPE_CHECKING, Mapping, Optional
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.linalg import (
+ Vectors,
+ Matrices,
+ DenseVector,
+ SparseVector,
+ DenseMatrix,
+ SparseMatrix,
+)
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+from pyspark.sql.connect.expressions import LiteralExpression
+
+if TYPE_CHECKING:
+ from pyspark.sql.connect.client import SparkConnectClient
+ from pyspark.ml.param import Params
+
+
+def serialize_param(value: Any, client: "SparkConnectClient") -> pb2.Param:
+ if isinstance(value, DenseVector):
+ return pb2.Param(vector=pb2.Vector(dense=pb2.Vector.Dense(value=value.values.tolist())))
+ elif isinstance(value, SparseVector):
+ return pb2.Param(
+ vector=pb2.Vector(
+ sparse=pb2.Vector.Sparse(
+ size=value.size, index=value.indices.tolist(), value=value.values.tolist()
+ )
+ )
+ )
+ elif isinstance(value, DenseMatrix):
+ return pb2.Param(
+ matrix=pb2.Matrix(
+ dense=pb2.Matrix.Dense(
+ num_rows=value.numRows, num_cols=value.numCols, value=value.values.tolist()
+ )
+ )
+ )
+ elif isinstance(value, SparseMatrix):
+ return pb2.Param(
+ matrix=pb2.Matrix(
+ sparse=pb2.Matrix.Sparse(
+ num_rows=value.numRows,
+ num_cols=value.numCols,
+ colptr=value.colPtrs.tolist(),
+ row_index=value.rowIndices.tolist(),
+ value=value.values.tolist(),
+ )
+ )
+ )
+ else:
+ literal = LiteralExpression._from_value(value).to_plan(client).literal
+ return pb2.Param(literal=literal)
+
+
+def serialize(client: "SparkConnectClient", *args: Any) -> List[Any]:
+ result = []
+ for arg in args:
+ if isinstance(arg, RemoteDataFrame):
+ result.append(pb2.FetchModelAttr.Args(input=arg._plan.plan(client)))
+ else:
+ result.append(pb2.FetchModelAttr.Args(param=serialize_param(arg, client)))
+ return result
+
+
+def deserialize_param(param: pb2.Param) -> Any:
+ if param.HasField("literal"):
+ return LiteralExpression._to_value(param.literal)
+ if param.HasField("vector"):
+ vector = param.vector
+ # TODO support sparse vector
+ if vector.HasField("dense"):
+ return Vectors.dense(vector.dense.value)
+ raise ValueError("TODO, support sparse vector")
+
+ if param.HasField("matrix"):
+ matrix = param.matrix
+ # TODO support sparse matrix
+ if matrix.HasField("dense") and not matrix.dense.is_transposed:
+ return Matrices.dense(
+ matrix.dense.num_rows,
+ matrix.dense.num_cols,
+ matrix.dense.value,
+ )
+ raise ValueError("TODO, support sparse matrix")
+
+ raise ValueError("Unsupported param type")
+
+
+def deserialize(ml_command_result: Optional[pb2.MlCommandResponse]) -> Any:
+ assert ml_command_result is not None
+ if ml_command_result.HasField("operator_info"):
+ return ml_command_result.operator_info
+
+ if ml_command_result.HasField("param"):
+ return deserialize_param(ml_command_result.param)
+ raise ValueError()
Review Comment:
Needs an error message.
##########
python/pyspark/ml/tests/test_persistence.py:
##########
@@ -153,29 +153,6 @@ def test_linear_regression_pmml_basic(self):
self.assertIn("Apache Spark", pmml_text)
self.assertIn("PMML", pmml_text)
- def test_logistic_regression(self):
Review Comment:
where did these tests move to?
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -1143,6 +1148,26 @@ def execute_command_as_iterator(
},
)
+ def execute_ml(self, req: pb2.ExecutePlanRequest) -> Optional[pb2.MlCommandResponse]:
+ """
+ Execute the ML command request and return ML response result
+ Parameters
+ ----------
+ req : pb2.ExecutePlanRequest
+ Proto representation of the plan.
+ """
+ logger.info("Execute ML")
+ try:
+ for attempt in self._retrying():
+ with attempt:
+ for b in self._stub.ExecutePlan(req, metadata=self._builder.metadata()):
+ if b.HasField("ml_command_result"):
+ return b.ml_command_result
Review Comment:
This is a bit tricky here as you don't fully consume the result iterator. In addition, this needs to be integrated with the progress handler that we have.
Wouldn't it be better to use the https://github.com/apache/spark/blob/master/python/pyspark/sql/connect/client/core.py#L1095 method?
##########
python/pyspark/ml/remote/util.py:
##########
@@ -0,0 +1,259 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import functools
+import os
+from typing import Any, cast, TypeVar, Callable, TYPE_CHECKING, Type
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, serialize, deserialize
+from pyspark.sql import is_remote
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+
+if TYPE_CHECKING:
+ from pyspark.ml.wrapper import JavaWrapper, JavaEstimator
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+
+FuncT = TypeVar("FuncT", bound=Callable[..., Any])
+
+
+def try_remote_intermediate_result(f: FuncT) -> FuncT:
+ """Mark the function/property that returns the intermediate result of the remote call.
+ Eg, model.summary"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper") -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ return f"{self._java_obj}.{f.__name__}"
+ else:
+ return f(self)
+
+ return cast(FuncT, wrapped)
+
+
+def try_remote_attribute_relation(f: FuncT) -> FuncT:
+ """Mark the function/property that returns a Relation.
+ Eg, model.summary.roc"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper", *args: Any, **kwargs: Any) -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ # The attribute returns a dataframe, we need to wrap it
+ # in the _ModelAttributeRelationPlan
+ from pyspark.ml.remote.proto import _ModelAttributeRelationPlan
+ from pyspark.sql.connect.session import SparkSession
+
+ assert isinstance(self._java_obj, str)
Review Comment:
Should we make this a more explicit error handling part of the constructor of the `_ModelAttributeRelationPlan`?
##########
python/pyspark/sql/connect/client/core.py:
##########
@@ -1143,6 +1148,26 @@ def execute_command_as_iterator(
},
)
+ def execute_ml(self, req: pb2.ExecutePlanRequest) -> Optional[pb2.MlCommandResponse]:
+ """
+ Execute the ML command request and return ML response result
+ Parameters
+ ----------
+ req : pb2.ExecutePlanRequest
+ Proto representation of the plan.
+ """
+ logger.info("Execute ML")
+ try:
+ for attempt in self._retrying():
+ with attempt:
+ for b in self._stub.ExecutePlan(req, metadata=self._builder.metadata()):
+ if b.HasField("ml_command_result"):
+ return b.ml_command_result
Review Comment:
Or basically we should model it similarly.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/execution/ExecuteThreadRunner.scala:
##########
@@ -311,6 +313,17 @@ private[connect] class ExecuteThreadRunner(executeHolder: ExecuteHolder) extends
planner.process(command = command, responseObserver = responseObserver)
}
+ private def handleMLCommand(request: proto.ExecutePlanRequest): Unit = {
Review Comment:
If we define the ML Command as a "kind" of command this becomes a bit easier and less burden on the ML side to manage the execution independently. You get all of the benefit of the infrastructure and it just works.
##########
sql/connect/common/src/main/protobuf/spark/connect/base.proto:
##########
@@ -384,6 +386,9 @@ message ExecutePlanResponse {
// Response for command that checkpoints a DataFrame.
CheckpointCommandResult checkpoint_command_result = 19;
+ // ML command response
+ MlCommandResponse ml_command_result = 100;
Review Comment:
This would move into the response type one of
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLCache.scala:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.ml
+
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.Model
+
+// TODO need to support persistence for model if memory is tight
Review Comment:
Let's file some sub jiras for this.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Model, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw new RuntimeException("Unsupported parameter type")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the fit command
+ *
+ * @param fit
+ * command
+ * @return
+ * an Estimator
+ */
+ def getEstimator(fit: proto.MlCommand.Fit): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the fit command
+ val name = fit.getEstimator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw new RuntimeException(s"Failed to find estimator: $name")
+ }
+ val uid = fit.getEstimator.getUid
+ val estimator: Estimator[_] = estimators(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Estimator[_]]
+
+ // Set parameters for the estimator
+ val params = fit.getParams
+ MLUtils.setInstanceParams(estimator, params)
+ estimator
+ }
+
+ def loadModel(className: String, path: String): Model[_] = {
+ // scalastyle:off classforname
+ val clazz = Class.forName(className)
+ // scalastyle:on classforname
+ val model = invokeStaticMethod(clazz, "load", path)
+ model.asInstanceOf[Model[_]]
+ }
+
+ /**
+ * Get the transformer instance according to the transform proto
+ *
+ * @param transformProto
+ * transform proto
+ * @return
+ * a Transformer
+ */
+ def getTransformer(transformProto: proto.MlRelation.Transform): Transformer = {
+ // Get the transformer name
+ val name = transformProto.getTransformer.getName
+ if (transformers.isEmpty || !transformers.contains(name)) {
+ throw new RuntimeException(s"Failed to find transformer: $name")
+ }
+ val uid = transformProto.getTransformer.getUid
+ val transformer = transformers(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Transformer]
+
+ val params = transformProto.getParams
+ MLUtils.setInstanceParams(transformer, params)
+ transformer
+ }
+
+ private lazy val ALLOWED_ATTRIBUTES = HashSet(
Review Comment:
This needs a bit more documentation.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.Model
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.MLUtils.loadModel
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+private class ModelAttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ private lazy val model = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(model.asInstanceOf[Object]) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(model.asInstanceOf[Object], lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(model.asInstanceOf[Object]) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchModelAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+object MLHandler extends Logging {
Review Comment:
This needs a lot more documentation to explain how it's used and what it does. Why can it be an object etc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876968390
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -97,13 +98,60 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
+ oneof ml_type {
+ Transform transform = 1;
+ FetchAttr fetch_attr = 2;
+ }
+ // Relation to represent transform(input) of the operator
+ // which could be a cached model or a new transformer
+ message Transform {
+ oneof operator {
+ // Object reference
+ ObjectRef obj_ref = 1;
+ // Could be an ML transformer like VectorAssembler
+ MlOperator transformer = 2;
+ }
+ // the input dataframe
+ Relation input = 3;
+ // the operator specific parameters
+ MlParams params = 4;
+ }
+}
+
+// Message for fetching attribute from object on the server side.
+// FetchAttr can be represented as a Relation or a ML command
+// Eg, model.coefficients, model.summary.weightedPrecision
+// or model.summary.roc which returns a DataFrame
+message FetchAttr {
Review Comment:
TBH this feels a bit too much as a free for all. I understand the need to fetch attributes, however as soon as we are passing arguments it feel like we are doing far more than just reading data. Why is this needed? Is there a way to make this more structured?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876933726
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the operator name
+ val name = operator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported Estimator, found ${name}")
+ }
+ val uid = operator.getUid
+ val estimator: Estimator[_] = estimators(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Estimator[_]]
+
+ // Set parameters for the estimator
+ params.foreach(p => MLUtils.setInstanceParams(estimator, p))
+ estimator
+ }
+
+ def load(className: String, path: String): Object = {
+ // scalastyle:off classforname
+ val clazz = Class.forName(className)
Review Comment:
Use `SparkClassUtils.classForName(..)`.
Also if this is Scala, you can also resolve the Module class and call load on it.
Other question, why are you not using the registry for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876942663
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the operator name
+ val name = operator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported Estimator, found ${name}")
+ }
+ val uid = operator.getUid
+ val estimator: Estimator[_] = estimators(name)
Review Comment:
Why no cache the constructor in the map... There is no reason to resolve it every time...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876956145
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/Serializer.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.linalg.{DenseMatrix, DenseVector, SparseMatrix, SparseVector}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.service.SessionHolder
+
+private[ml] object Serializer {
+
+ /**
+ * Serialize the ML parameters, currently only support Vector/Matrix and literals
Review Comment:
Do you expect other types?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1881450556
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
Review Comment:
We indeed expect to support other 3rd-party library for ML on the connect in the follow up.
> We are using fully qualified names for resolution. In that case you don't really need the service loader because you can just use SparkClassUtils.classForName(..). Are there plans to deviate from this?
Wow, thx, good to know that. Hi @grundprinzip, any idea about not using ServiceLoader?
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
Review Comment:
Sounds good
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
Review Comment:
Wow, good catch. Currently, for the existing LogisticRegression, I didn't see the Long parameters. So looks like this function is not tested.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
Review Comment:
Make the function into an inner function, so the caller will ensure the type is matching.
##########
sql/connect/common/src/main/protobuf/spark/connect/ml_common.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// MlParams stores param settings for ML Estimator / Transformer / Evaluator
+message MlParams {
+ // User-supplied params
+ map<string, Param> params = 1;
+}
+
+// Represents the parameter type of the ML instance, or the returned value
+// of the attribute
+message Param {
+ oneof param_type {
+ Expression.Literal literal = 1;
+ Vector vector = 2;
Review Comment:
The Matrix/Vector are really ML-specifics, do you really want them to be supported in literals?
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
Review Comment:
Added a scala test for it.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
Review Comment:
This method chain is an optimization for the special case in ML when accessing "summary" in a model, which returns a corresponding Summary class in python. And then we would access the attributes of the Summary in python further. So we could use "method chains" like "model_id.summary.accuracy" not only reduce the gRPC calls, but also not cache the summary on the server side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2588867463
had offline discussion with @wbo4958 , separate messages for vector/matrix are compact so more suitable for large params like initial model weights.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2586104252
> Hi @zhengruifeng,
>
> > I also think we can optimize out the message Param:
> > option 1: support Vector and Matrix in message Literal;
> > option 2 (Preferred): Using message Expression or Expression.Literal.Struct to express the parameters;
>
> If you insist on optimizing it, I would like to add vector/matrix in the Literal. What's your opinion?
Actually, I personally prefer option 2 (Using `Expression.Literal.Struct`) to minimize the protobuf changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1915951762
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
Review Comment:
Is this used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2499835314
Hey @wbo4958 , I just wanted to express my Kudos for your work. I tried your PR locally and wanted to see how much work it is to add an additional feature transformer. And it turns out it's exactly one line of code!
Thank you so much for your effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876703045
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
Review Comment:
Not a big problem: This prevents users from plugging an ML library through either addJar or addArtifact later on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876912227
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
Review Comment:
So you are reconciling the parameter value. `convert` is a bit misleading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1877882367
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -111,6 +112,9 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
private[spark] lazy val dataFrameCache: ConcurrentMap[String, DataFrame] =
new ConcurrentHashMap()
+ // ML model cache
+ private[connect] lazy val mlCache = new MLCache()
Review Comment:
Currently, I have no idea about the size of the objects, some object should be small, while some are large, it depends.
Yes, the de-constructor of the python class will trigger a GRPC to delete the cache on the server side.
##########
mllib/src/main/scala/org/apache/spark/ml/util/Summary.scala:
##########
@@ -0,0 +1,26 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.ml.util
+
+import org.apache.spark.annotation.Since
+
+/**
+ * Trait for the Summary
+ */
+@Since("4.0.0")
+private[spark] trait Summary
Review Comment:
Make all the summaries extend from this Summary, so in connect, we can match to this trait and decide if the object of the summary needs to be cached.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLCache.scala:
##########
@@ -0,0 +1,70 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.connect.ml
+
+import java.util.UUID
+import java.util.concurrent.ConcurrentHashMap
+
+import org.apache.spark.internal.Logging
+
+/**
+ * MLCache is for caching ML objects, typically for models and summaries evaluated by a model.
+ */
+private[connect] class MLCache extends Logging {
+ private val cachedModel: ConcurrentHashMap[String, Object] =
+ new ConcurrentHashMap[String, Object]()
+
+ /**
+ * Cache an object into a map of MLCache, and return its key
+ * @param obj
+ * the object to be cached
+ * @return
+ * the key
+ */
+ def register(obj: Object): String = {
+ val objectId = UUID.randomUUID().toString.takeRight(12)
Review Comment:
Hmm, we could use the full UUID string instead of just taking the 12 chars from right.
> An alternative would be to generate a predictable Id.
Could you show me how to do that?
##########
sql/connect/common/src/main/protobuf/spark/connect/ml_common.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// MlParams stores param settings for ML Estimator / Transformer / Evaluator
+message MlParams {
+ // User-supplied params
+ map<string, Param> params = 1;
+}
+
+// Represents the parameter type of the ML instance, or the returned value
+// of the attribute
+message Param {
+ oneof param_type {
+ Expression.Literal literal = 1;
+ Vector vector = 2;
+ Matrix matrix = 3;
+ }
+}
+
+// MLOperator represents the ML operators like (Estimator, Transformer or Evaluator)
+message MlOperator {
+ // The qualified name of the ML operator.
+ string name = 1;
+ // Unique id of the ML operator
+ string uid = 2;
Review Comment:
No, the UID is the Identifiable for each ML operators.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
Review Comment:
Sounds good. Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1914210183
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters._
+
+import org.apache.commons.lang3.reflect.MethodUtils.invokeMethod
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.{SparkClassUtils, Utils}
+
+private[ml] object MLUtils {
+
+ /**
+ * Load the registered ML operators via ServiceLoader
+ *
+ * @param mlCls
+ * the operator class
+ * @return
+ * a Map with name and class
+ */
+ private def loadOperators(mlCls: Class[_]): Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(mlCls, loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val estimators = loadOperators(classOf[Estimator[_]])
+
+ private lazy val transformers = loadOperators(classOf[Transformer])
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ /**
+ * Set the parameters to the ML instance
+ *
+ * @param instance
+ * an ML operator
+ * @param params
+ * the parameters of the ML operator
+ */
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ reconcileParam(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ /**
+ * Convert the array from Object[] to Array[_]
+ * @param elementType
+ * the element type of the array
+ * @param array
+ * to be reconciled
+ * @return
+ * the reconciled array
+ */
+ private def reconcileArray(elementType: Class[_], array: Array[_]): Array[_] = {
+ if (elementType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (elementType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (elementType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (elementType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (elementType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (elementType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (elementType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ throw MlUnsupportedException(
+ s"array element type unsupported, " +
+ s"found ${elementType.getName}")
+ }
+ }
+
+ /**
+ * Reconcile the parameter value given the provided parameter type. Currently, support
+ * byte/short/int/long/float/double/string and array. Note that, array of array is not supported
+ * yet.
+ */
+ private def reconcileParam(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType == classOf[Boolean]) {
+ value.asInstanceOf[Boolean]
+ } else if (paramType == classOf[String]) {
+ value.asInstanceOf[String]
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ if (compType.isArray) {
+ throw MlUnsupportedException(s"Array of array unsupported")
+ } else {
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ reconcileParam(compType, e)
+ }
+ reconcileArray(compType, array)
+ }
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type, found ${paramType.getName}")
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the instance according to the provided proto information.
+ *
+ * @param name
+ * The name of the instance (either estimator or transformer).
+ * @param uid
+ * The unique identifier for the instance.
+ * @param instanceMap
+ * A map of instance names to constructors.
+ * @param params
+ * Optional parameters for the instance.
+ * @tparam T
+ * The type of the instance (Estimator or Transformer).
+ * @return
+ * The instance of the requested type.
+ * @throws MlUnsupportedException
+ * If the instance is not supported.
+ */
+ private def getInstance[T](
+ name: String,
+ uid: String,
+ instanceMap: Map[String, Class[_]],
+ params: Option[proto.MlParams]): T = {
+ if (instanceMap.isEmpty || !instanceMap.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported ML operator, found $name")
+ }
+
+ val instance = instanceMap(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[T]
+
+ // Set parameters for the instance if they are provided
+ params.foreach(p => MLUtils.setInstanceParams(instance.asInstanceOf[Params], p))
+ instance
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ val name = operator.getName
+ val uid = operator.getUid
+ getInstance[Estimator[_]](name, uid, estimators, params)
+ }
+
+ /**
+ * Get the transformer instance according to the transform proto
+ *
+ * @param transformProto
+ * transform proto
+ * @return
+ * a transformer
+ */
+ def getTransformer(transformProto: proto.MlRelation.Transform): Transformer = {
+ val name = transformProto.getTransformer.getName
+ val uid = transformProto.getTransformer.getUid
+ val params = transformProto.getParams
+ getInstance[Transformer](name, uid, transformers, Some(params))
+ }
+
+ /**
+ * Call "load: function on the ML operator given the operator name
+ *
+ * @param className
+ * the ML operator name
+ * @param path
+ * the path to be loaded
+ * @return
+ * the ML instance
+ */
+ def load(className: String, path: String): Object = {
+ val loadedMethod = SparkClassUtils.classForName(className).getMethod("load", classOf[String])
+ loadedMethod.invoke(null, path)
+ }
+
+ // Since we're using reflection way to get the attribute, in order not to
+ // leave a security hole, we define an allowed attribute list that can be accessed.
+ // The attributes could be retrieved from the corresponding python class
+ private lazy val ALLOWED_ATTRIBUTES = HashSet(
+ "toString",
+ "numFeatures",
+ "predict", // PredictionModel
+ "numClasses",
+ "predictRaw", // ClassificationModel
+ "predictProbability", // ProbabilisticClassificationModel
+ "coefficients",
+ "intercept",
+ "coefficientMatrix",
+ "interceptVector", // LogisticRegressionModel
+ "summary",
+ "hasSummary",
+ "evaluate", // LogisticRegressionModel
+ "predictions",
+ "predictionCol",
+ "labelCol",
+ "weightCol",
+ "labels", // _ClassificationSummary
+ "truePositiveRateByLabel",
+ "falsePositiveRateByLabel", // _ClassificationSummary
+ "precisionByLabel",
+ "recallByLabel",
+ "fMeasureByLabel",
+ "accuracy", // _ClassificationSummary
+ "weightedTruePositiveRate",
+ "weightedFalsePositiveRate", // _ClassificationSummary
+ "weightedRecall",
+ "weightedPrecision",
+ "weightedFMeasure", // _ClassificationSummary
+ "scoreCol",
+ "roc",
+ "areaUnderROC",
+ "pr",
+ "fMeasureByThreshold", // _BinaryClassificationSummary
+ "precisionByThreshold",
+ "recallByThreshold", // _BinaryClassificationSummary
+ "probabilityCol",
+ "featuresCol", // LogisticRegressionSummary
+ "objectiveHistory",
+ "totalIterations" // _TrainingSummary
+ )
+
+ def invokeMethodAllowed(obj: Object, methodName: String): Object = {
+ if (!ALLOWED_ATTRIBUTES.contains(methodName)) {
Review Comment:
we should add tests for the reflection part, but it is ok to do it in follow-up PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1882376858
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
Review Comment:
I'm +1 on grouping the commands as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1853195049
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,234 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.Model
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.MLUtils.loadModel
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+private class ModelAttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ private lazy val model = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(model.asInstanceOf[Object]) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(model.asInstanceOf[Object], lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(model.asInstanceOf[Object]) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchModelAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+object MLHandler extends Logging {
Review Comment:
Added some documentation like "MLHandler is a utility to group all ML operations".
> Why can it be an object
Not have a strong reason it must be an object. Any idea if moving it to a class?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1881587202
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the operator name
+ val name = operator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported Estimator, found ${name}")
+ }
+ val uid = operator.getUid
+ val estimator: Estimator[_] = estimators(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Estimator[_]]
+
+ // Set parameters for the estimator
+ params.foreach(p => MLUtils.setInstanceParams(estimator, p))
+ estimator
+ }
+
+ def load(className: String, path: String): Object = {
+ // scalastyle:off classforname
+ val clazz = Class.forName(className)
Review Comment:
Oops, for Model we couldn't use registry since there's no default non-param constructor there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907] Support spark.ml on Connect [spark]
Posted by "michTalebzadeh (via GitHub)" <gi...@apache.org>.
michTalebzadeh commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2473588444
You point
t's a new feature that makes spark.ml run on connect environment.
Can you please explain more on why this feature which already exists on Spark itself, will be beneficial on connect?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "WeichenXu123 (via GitHub)" <gi...@apache.org>.
WeichenXu123 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1860583738
##########
python/pyspark/ml/remote/proto.py:
##########
@@ -0,0 +1,86 @@
+#
Review Comment:
agreed that we'd better move it to the right package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2499883841
Sorry, clicked the wrong button!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1883375109
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
Review Comment:
Thx, the new commit has refactored the "Fetch" a little bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1877543825
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/Serializer.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.linalg.{DenseMatrix, DenseVector, SparseMatrix, SparseVector}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.service.SessionHolder
+
+private[ml] object Serializer {
+
+ /**
+ * Serialize the ML parameters, currently only support Vector/Matrix and literals
Review Comment:
No, for now, I haven't seen other types that need to be supported for spark.ml
But probably we need when we're supporting more estimators.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1914230923
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,353 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters._
+
+import org.apache.commons.lang3.reflect.MethodUtils.invokeMethod
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.{SparkClassUtils, Utils}
+
+private[ml] object MLUtils {
+
+ /**
+ * Load the registered ML operators via ServiceLoader
+ *
+ * @param mlCls
+ * the operator class
+ * @return
+ * a Map with name and class
+ */
+ private def loadOperators(mlCls: Class[_]): Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(mlCls, loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val estimators = loadOperators(classOf[Estimator[_]])
+
+ private lazy val transformers = loadOperators(classOf[Transformer])
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ /**
+ * Set the parameters to the ML instance
+ *
+ * @param instance
+ * an ML operator
+ * @param params
+ * the parameters of the ML operator
+ */
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ reconcileParam(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ /**
+ * Convert the array from Object[] to Array[_]
+ * @param elementType
+ * the element type of the array
+ * @param array
+ * to be reconciled
+ * @return
+ * the reconciled array
+ */
+ private def reconcileArray(elementType: Class[_], array: Array[_]): Array[_] = {
+ if (elementType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (elementType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (elementType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (elementType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (elementType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (elementType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (elementType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ throw MlUnsupportedException(
+ s"array element type unsupported, " +
+ s"found ${elementType.getName}")
+ }
+ }
+
+ /**
+ * Reconcile the parameter value given the provided parameter type. Currently, support
+ * byte/short/int/long/float/double/string and array. Note that, array of array is not supported
+ * yet.
+ */
+ private def reconcileParam(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType == classOf[Boolean]) {
+ value.asInstanceOf[Boolean]
+ } else if (paramType == classOf[String]) {
+ value.asInstanceOf[String]
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ if (compType.isArray) {
+ throw MlUnsupportedException(s"Array of array unsupported")
+ } else {
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ reconcileParam(compType, e)
+ }
+ reconcileArray(compType, array)
+ }
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type, found ${paramType.getName}")
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the instance according to the provided proto information.
+ *
+ * @param name
+ * The name of the instance (either estimator or transformer).
+ * @param uid
+ * The unique identifier for the instance.
+ * @param instanceMap
+ * A map of instance names to constructors.
+ * @param params
+ * Optional parameters for the instance.
+ * @tparam T
+ * The type of the instance (Estimator or Transformer).
+ * @return
+ * The instance of the requested type.
+ * @throws MlUnsupportedException
+ * If the instance is not supported.
+ */
+ private def getInstance[T](
+ name: String,
+ uid: String,
+ instanceMap: Map[String, Class[_]],
+ params: Option[proto.MlParams]): T = {
+ if (instanceMap.isEmpty || !instanceMap.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported ML operator, found $name")
+ }
+
+ val instance = instanceMap(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[T]
+
+ // Set parameters for the instance if they are provided
+ params.foreach(p => MLUtils.setInstanceParams(instance.asInstanceOf[Params], p))
+ instance
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ val name = operator.getName
+ val uid = operator.getUid
+ getInstance[Estimator[_]](name, uid, estimators, params)
+ }
+
+ /**
+ * Get the transformer instance according to the transform proto
+ *
+ * @param transformProto
+ * transform proto
+ * @return
+ * a transformer
+ */
+ def getTransformer(transformProto: proto.MlRelation.Transform): Transformer = {
+ val name = transformProto.getTransformer.getName
+ val uid = transformProto.getTransformer.getUid
+ val params = transformProto.getParams
+ getInstance[Transformer](name, uid, transformers, Some(params))
+ }
+
+ /**
+ * Call "load: function on the ML operator given the operator name
+ *
+ * @param className
+ * the ML operator name
+ * @param path
+ * the path to be loaded
+ * @return
+ * the ML instance
+ */
+ def load(className: String, path: String): Object = {
+ val loadedMethod = SparkClassUtils.classForName(className).getMethod("load", classOf[String])
+ loadedMethod.invoke(null, path)
+ }
+
+ // Since we're using reflection way to get the attribute, in order not to
+ // leave a security hole, we define an allowed attribute list that can be accessed.
+ // The attributes could be retrieved from the corresponding python class
+ private lazy val ALLOWED_ATTRIBUTES = HashSet(
+ "toString",
+ "numFeatures",
+ "predict", // PredictionModel
+ "numClasses",
+ "predictRaw", // ClassificationModel
+ "predictProbability", // ProbabilisticClassificationModel
+ "coefficients",
+ "intercept",
+ "coefficientMatrix",
+ "interceptVector", // LogisticRegressionModel
+ "summary",
+ "hasSummary",
+ "evaluate", // LogisticRegressionModel
+ "predictions",
+ "predictionCol",
+ "labelCol",
+ "weightCol",
+ "labels", // _ClassificationSummary
+ "truePositiveRateByLabel",
+ "falsePositiveRateByLabel", // _ClassificationSummary
+ "precisionByLabel",
+ "recallByLabel",
+ "fMeasureByLabel",
+ "accuracy", // _ClassificationSummary
+ "weightedTruePositiveRate",
+ "weightedFalsePositiveRate", // _ClassificationSummary
+ "weightedRecall",
+ "weightedPrecision",
+ "weightedFMeasure", // _ClassificationSummary
+ "scoreCol",
+ "roc",
+ "areaUnderROC",
+ "pr",
+ "fMeasureByThreshold", // _BinaryClassificationSummary
+ "precisionByThreshold",
+ "recallByThreshold", // _BinaryClassificationSummary
+ "probabilityCol",
+ "featuresCol", // LogisticRegressionSummary
+ "objectiveHistory",
+ "totalIterations" // _TrainingSummary
+ )
+
+ def invokeMethodAllowed(obj: Object, methodName: String): Object = {
+ if (!ALLOWED_ATTRIBUTES.contains(methodName)) {
Review Comment:
Yeah, the follow up PR will add the tests for reflection. Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "LuciferYang (via GitHub)" <gi...@apache.org>.
LuciferYang commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1915981028
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
Review Comment:
Seems unused, I found `spark/connect/ml.proto:22:1: warning: Import spark/connect/expressions.proto is unused.` in compile log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2574315959
Hi @martin-g, @hvanhovell, Would you help review it again. Thx very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876703850
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
Review Comment:
Let's try to put this in a helper function so you can share this with `transformers`...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "WeichenXu123 (via GitHub)" <gi...@apache.org>.
WeichenXu123 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1842006124
##########
python/pyspark/ml/remote/util.py:
##########
@@ -0,0 +1,238 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import functools
+import os
+from typing import Any, cast, TypeVar, Callable, TYPE_CHECKING, Type
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, serialize, deserialize
+from pyspark.sql import is_remote
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+
+if TYPE_CHECKING:
+ from pyspark.ml.wrapper import JavaWrapper, JavaEstimator
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+
+FuncT = TypeVar("FuncT", bound=Callable[..., Any])
+
+
+def try_remote_intermediate_result(f: FuncT) -> FuncT:
+ """Mark the function/property that returns the intermediate result of the remote call.
+ Eg, model.summary"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper") -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ return f"{self._java_obj}.{f.__name__}"
Review Comment:
how does it work here ? We need to send RPC request to spark server get the attribute value
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876859041
##########
sql/connect/common/src/main/protobuf/spark/connect/ml_common.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// MlParams stores param settings for ML Estimator / Transformer / Evaluator
+message MlParams {
+ // User-supplied params
+ map<string, Param> params = 1;
+}
+
+// Represents the parameter type of the ML instance, or the returned value
+// of the attribute
+message Param {
+ oneof param_type {
+ Expression.Literal literal = 1;
+ Vector vector = 2;
Review Comment:
Shouldn't we make Vector/Matrix supported literals?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876867462
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
Review Comment:
I am sorry if I am being dense. Where is predict/transform? I can see that everything else in the model life cycle is present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876944337
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the operator name
+ val name = operator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported Estimator, found ${name}")
+ }
+ val uid = operator.getUid
+ val estimator: Estimator[_] = estimators(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Estimator[_]]
+
+ // Set parameters for the estimator
+ params.foreach(p => MLUtils.setInstanceParams(estimator, p))
+ estimator
+ }
+
+ def load(className: String, path: String): Object = {
+ // scalastyle:off classforname
+ val clazz = Class.forName(className)
+ // scalastyle:on classforname
+ invokeStaticMethod(clazz, "load", path)
+ }
+
+ /**
+ * Get the transformer instance according to the transform proto
+ *
+ * @param transformProto
+ * transform proto
+ * @return
+ * a transformer
+ */
+ def getTransformer(transformProto: proto.MlRelation.Transform): Transformer = {
+ // Get the transformer name
Review Comment:
This is basically the same code as getEstimator, you could try to factor this into a helper.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876927504
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
Review Comment:
`paramType` should be `elementType` or `componentType`.
Also, I am not entirely sure this code does what we expect it to do. At best this code changes the `componentType` of the array. It does not cast the datatypes. For example this code will not work:
```scala
val elements = Array.tabulate(10)(i => i.asInstanceOf[Any])
// Cast to int works
elements.map(_.asInstanceOf[Int])
>> val res23: Array[Int] = Array(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
// Cast to long fails
elements.map(_.asInstanceOf[Long])
>> java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Long (java.lang.Integer and java.lang.Long are in module java.base of loader 'bootstrap')
>> at scala.runtime.BoxesRunTime.unboxToLong(BoxesRunTime.java:103)
>> at $anonfun$res24$1(<console>:1)
>> at $anonfun$res24$1$adapted(<console>:1)
>> at scala.collection.ArrayOps$.map$extension(ArrayOps.scala:935)
>> ... 42 elided
```
Do you have a test for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876852174
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/service/SessionHolder.scala:
##########
@@ -111,6 +112,9 @@ case class SessionHolder(userId: String, sessionId: String, session: SparkSessio
private[spark] lazy val dataFrameCache: ConcurrentMap[String, DataFrame] =
new ConcurrentHashMap()
+ // ML model cache
+ private[connect] lazy val mlCache = new MLCache()
Review Comment:
Is there a way to understand the size of the objects stored in the MLCache? Is the python session pro-actively cleaning up the object?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1877028075
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ protected lazy val instance = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(instance, lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+}
+
+private class ModelAttributeHelper(
+ sessionHolder: SessionHolder,
+ objIdentifier: String,
+ method: Option[String],
+ argValues: Array[Object] = Array.empty,
+ argClasses: Array[Class[_]] = Array.empty)
+ extends AttributeHelper(sessionHolder, objIdentifier, method, argValues, argClasses) {
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val model = instance.asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object AttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): AttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new AttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+// MLHandler is a utility to group all ML operations
+object MLHandler extends Logging {
+ def handleMlCommand(
+ sessionHolder: SessionHolder,
+ mlCommand: proto.MlCommand): proto.MlCommandResult = {
+
+ val mlCache = sessionHolder.mlCache
+
+ mlCommand.getCommandCase match {
+ case proto.MlCommand.CommandCase.FIT =>
+ val fitCmd = mlCommand.getFit
+ val estimatorProto = fitCmd.getEstimator
+ assert(estimatorProto.getType == proto.MlOperator.OperatorType.ESTIMATOR)
+
+ val dataset = MLUtils.parseRelationProto(fitCmd.getDataset, sessionHolder)
+ val estimator = MLUtils.getEstimator(estimatorProto, Some(fitCmd.getParams))
+ val model = estimator.fit(dataset).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id)))
+ .build()
+
+ case proto.MlCommand.CommandCase.FETCH_ATTR =>
+ val args = mlCommand.getFetchAttr.getArgsList.asScala.toArray
+ val helper = AttributeHelper(
+ sessionHolder,
+ mlCommand.getFetchAttr.getObjRef.getId,
+ Option(mlCommand.getFetchAttr.getMethod),
+ args)
+ val attrResult = helper.getAttribute
+ attrResult match {
+ case s: Summary =>
+ val id = mlCache.register(s)
+ proto.MlCommandResult.newBuilder().setSummary(id).build()
+ case _ =>
+ val param = Serializer.serializeParam(attrResult)
+ proto.MlCommandResult.newBuilder().setParam(param).build()
+ }
+
+ case proto.MlCommand.CommandCase.DELETE =>
+ val modelId = mlCommand.getDelete.getObjRef.getId
+ var result = false
+ if (!modelId.contains(".")) {
Review Comment:
Why is "." excluded?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1877444156
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ protected lazy val instance = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(instance, lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+}
+
+private class ModelAttributeHelper(
+ sessionHolder: SessionHolder,
+ objIdentifier: String,
+ method: Option[String],
+ argValues: Array[Object] = Array.empty,
+ argClasses: Array[Class[_]] = Array.empty)
+ extends AttributeHelper(sessionHolder, objIdentifier, method, argValues, argClasses) {
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val model = instance.asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object AttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): AttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new AttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+// MLHandler is a utility to group all ML operations
+object MLHandler extends Logging {
+ def handleMlCommand(
+ sessionHolder: SessionHolder,
+ mlCommand: proto.MlCommand): proto.MlCommandResult = {
+
+ val mlCache = sessionHolder.mlCache
+
+ mlCommand.getCommandCase match {
+ case proto.MlCommand.CommandCase.FIT =>
+ val fitCmd = mlCommand.getFit
+ val estimatorProto = fitCmd.getEstimator
+ assert(estimatorProto.getType == proto.MlOperator.OperatorType.ESTIMATOR)
+
+ val dataset = MLUtils.parseRelationProto(fitCmd.getDataset, sessionHolder)
+ val estimator = MLUtils.getEstimator(estimatorProto, Some(fitCmd.getParams))
+ val model = estimator.fit(dataset).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id)))
+ .build()
+
+ case proto.MlCommand.CommandCase.FETCH_ATTR =>
+ val args = mlCommand.getFetchAttr.getArgsList.asScala.toArray
+ val helper = AttributeHelper(
+ sessionHolder,
+ mlCommand.getFetchAttr.getObjRef.getId,
+ Option(mlCommand.getFetchAttr.getMethod),
+ args)
+ val attrResult = helper.getAttribute
+ attrResult match {
+ case s: Summary =>
+ val id = mlCache.register(s)
+ proto.MlCommandResult.newBuilder().setSummary(id).build()
+ case _ =>
+ val param = Serializer.serializeParam(attrResult)
+ proto.MlCommandResult.newBuilder().setParam(param).build()
+ }
+
+ case proto.MlCommand.CommandCase.DELETE =>
+ val modelId = mlCommand.getDelete.getObjRef.getId
+ var result = false
+ if (!modelId.contains(".")) {
+ mlCache.remove(modelId)
+ result = true
+ }
+ proto.MlCommandResult
+ .newBuilder()
+ .setParam(
+ proto.Param
+ .newBuilder()
+ .setLiteral(LiteralValueProtoConverter.toLiteralProto(result))
+ .build())
+ .build()
+
+ case proto.MlCommand.CommandCase.WRITE =>
+ mlCommand.getWrite.getTypeCase match {
+ case proto.MlCommand.Writer.TypeCase.OBJ_REF => // save a model
+ val modelId = mlCommand.getWrite.getObjRef.getId
+ val model = mlCache.get(modelId).asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, mlCommand.getWrite.getParams)
+
+ copiedModel match {
+ case m: MLWritable => MLUtils.write(m, mlCommand.getWrite)
+ case other => throw MlUnsupportedException(s"$other is not writable")
+ }
+
+ // save an estimator/evaluator/transformer
+ case proto.MlCommand.Writer.TypeCase.OPERATOR =>
+ val writer = mlCommand.getWrite
+ if (writer.getOperator.getType == proto.MlOperator.OperatorType.ESTIMATOR) {
+ val estimator = MLUtils.getEstimator(writer.getOperator, Some(writer.getParams))
+ estimator match {
+ case m: MLWritable => MLUtils.write(m, mlCommand.getWrite)
+ case other => throw MlUnsupportedException(s"Estimator $other is not writable")
+ }
+ } else {
+ throw MlUnsupportedException(s"${writer.getOperator.getName} not supported")
+ }
+
+ case other => throw MlUnsupportedException(s"$other not supported")
+ }
+ proto.MlCommandResult.newBuilder().build()
+
+ case proto.MlCommand.CommandCase.READ =>
+ val operator = mlCommand.getRead.getOperator
+ val name = operator.getName
+ val path = mlCommand.getRead.getPath
+
+ if (operator.getType == proto.MlOperator.OperatorType.MODEL) {
+ val model = MLUtils.load(name, path).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id))
+ .setUid(model.uid)
+ .setParams(Serializer.serializeParams(model)))
+ .build()
+
+ } else if (operator.getType == proto.MlOperator.OperatorType.ESTIMATOR) {
+ val estimator = MLUtils.load(name, path).asInstanceOf[Estimator[_]]
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setName(name)
+ .setUid(estimator.uid)
+ .setParams(Serializer.serializeParams(estimator)))
+ .build()
+ } else {
+ throw MlUnsupportedException(s"${operator.getType} not supported")
+ }
+
+ case other => throw MlUnsupportedException(s"$other not supported")
+ }
+ }
+
+ def transformMLRelation(relation: proto.MlRelation, sessionHolder: SessionHolder): DataFrame = {
Review Comment:
Hmm, This function indeed returns the Dataframe, but the caller will return its logicalPlan from https://github.com/apache/spark/pull/48791/files#diff-db5ed413eb10608faef1340838a236cbbac5ca71ad5193a61cc953c66a01c12fR234
``` scala
MLHandler.transformMLRelation(rel.getMlRelation, sessionHolder).logicalPlan
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1872325769
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,283 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ * @param sessionHolder
+ * @param objIdentifier
+ * @param method
+ * @param argValues
+ * @param argClasses
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ protected lazy val instance = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(instance, lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+}
+
+private class ModelAttributeHelper(
+ sessionHolder: SessionHolder,
+ objIdentifier: String,
+ method: Option[String],
+ argValues: Array[Object] = Array.empty,
+ argClasses: Array[Class[_]] = Array.empty)
+ extends AttributeHelper(sessionHolder, objIdentifier, method, argValues, argClasses) {
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val model = instance.asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object AttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): AttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new AttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+// MLHandler is a utility to group all ML operations
+object MLHandler extends Logging {
+ def handleMlCommand(
+ sessionHolder: SessionHolder,
+ mlCommand: proto.MlCommand): proto.MlCommandResult = {
+
+ val mlCache = sessionHolder.mlCache
+
+ mlCommand.getCommandCase match {
+ case proto.MlCommand.CommandCase.FIT =>
+ val fitCmd = mlCommand.getFit
+ val estimatorProto = fitCmd.getEstimator
+ assert(estimatorProto.getType == proto.MlOperator.OperatorType.ESTIMATOR)
+
+ val dataset = MLUtils.parseRelationProto(fitCmd.getDataset, sessionHolder)
+ val estimator = MLUtils.getEstimator(estimatorProto, Some(fitCmd.getParams))
+ val model = estimator.fit(dataset).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id)))
+ .build()
+
+ case proto.MlCommand.CommandCase.FETCH_ATTR =>
+ val args = mlCommand.getFetchAttr.getArgsList.asScala.toArray
+ val helper = AttributeHelper(
+ sessionHolder,
+ mlCommand.getFetchAttr.getObjRef.getId,
+ Option(mlCommand.getFetchAttr.getMethod),
+ args)
+ val attrResult = helper.getAttribute
+ attrResult match {
+ case s: Summary =>
+ val id = mlCache.register(s)
+ proto.MlCommandResult.newBuilder().setSummary(id).build()
+ case _ =>
+ val param = Serializer.serializeParam(attrResult)
+ proto.MlCommandResult.newBuilder().setParam(param).build()
+ }
+
+ case proto.MlCommand.CommandCase.DELETE =>
+ val modelId = mlCommand.getDelete.getObjRef.getId
+ var result = false
+ if (!modelId.contains(".")) {
+ mlCache.remove(modelId)
+ result = true
+ }
+ proto.MlCommandResult
+ .newBuilder()
+ .setParam(
+ proto.Param
+ .newBuilder()
+ .setLiteral(LiteralValueProtoConverter.toLiteralProto(result))
+ .build())
+ .build()
+
+ case proto.MlCommand.CommandCase.WRITE =>
+ mlCommand.getWrite.getTypeCase match {
+ case proto.MlCommand.Writer.TypeCase.OBJ_REF => // save a model
+ val modelId = mlCommand.getWrite.getObjRef.getId
+ val model = mlCache.get(modelId).asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, mlCommand.getWrite.getParams)
+
+ copiedModel match {
+ case m: MLWritable => MLUtils.write(m, mlCommand.getWrite)
+ case _ => throw new RuntimeException("Failed to handle model.save")
Review Comment:
Thx for the comment. Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2521744672
> I think I'm good with the current state of the PR and am looking for the next one that adds the next batch of operators and can maybe used for other contributors as well as a template.
>
> Please address my comment on the topic of error messages, but after that I'm ok with approving the PR.
>
> We can clean up the Python package names and locations in a follow up.
Sounds good, Thx @grundprinzip
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876966173
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -97,13 +98,60 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
Review Comment:
What is the benefit of adding MLRelation? Why not add the two MLRelations to the Relations message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1877543825
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/Serializer.scala:
##########
@@ -0,0 +1,151 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.linalg.{DenseMatrix, DenseVector, SparseMatrix, SparseVector}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.sql.Dataset
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.service.SessionHolder
+
+private[ml] object Serializer {
+
+ /**
+ * Serialize the ML parameters, currently only support Vector/Matrix and literals
Review Comment:
No, for now, I haven't seen other types that need to be supported for spark.ml
But probably we need after we're supporting other estimators.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the operator name
+ val name = operator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported Estimator, found ${name}")
+ }
+ val uid = operator.getUid
+ val estimator: Estimator[_] = estimators(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Estimator[_]]
+
+ // Set parameters for the estimator
+ params.foreach(p => MLUtils.setInstanceParams(estimator, p))
+ estimator
+ }
+
+ def load(className: String, path: String): Object = {
+ // scalastyle:off classforname
+ val clazz = Class.forName(className)
+ // scalastyle:on classforname
+ invokeStaticMethod(clazz, "load", path)
+ }
+
+ /**
+ * Get the transformer instance according to the transform proto
+ *
+ * @param transformProto
+ * transform proto
+ * @return
+ * a transformer
+ */
+ def getTransformer(transformProto: proto.MlRelation.Transform): Transformer = {
+ // Get the transformer name
Review Comment:
sounds good. Done.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the operator name
+ val name = operator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported Estimator, found ${name}")
+ }
+ val uid = operator.getUid
+ val estimator: Estimator[_] = estimators(name)
+ .getConstructor(classOf[String])
+ .newInstance(uid)
+ .asInstanceOf[Estimator[_]]
+
+ // Set parameters for the estimator
+ params.foreach(p => MLUtils.setInstanceParams(estimator, p))
+ estimator
+ }
+
+ def load(className: String, path: String): Object = {
+ // scalastyle:off classforname
+ val clazz = Class.forName(className)
Review Comment:
Thx for the suggestion about `SparkClassUtils.classForName`.
> Other question, why are you not using the registry for this.
Good catch, it seems we need to use the registry.
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -97,13 +98,60 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
Review Comment:
Not too much, just group the ML things together. The existing relations are SQL related. I just didn't want to mix ML things into SQL. But definitely, we could put them in the Relations.
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -97,13 +98,60 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
+ oneof ml_type {
+ Transform transform = 1;
+ FetchAttr fetch_attr = 2;
+ }
+ // Relation to represent transform(input) of the operator
+ // which could be a cached model or a new transformer
+ message Transform {
+ oneof operator {
+ // Object reference
+ ObjectRef obj_ref = 1;
+ // Could be an ML transformer like VectorAssembler
+ MlOperator transformer = 2;
+ }
+ // the input dataframe
+ Relation input = 3;
+ // the operator specific parameters
+ MlParams params = 4;
+ }
+}
+
+// Message for fetching attribute from object on the server side.
+// FetchAttr can be represented as a Relation or a ML command
+// Eg, model.coefficients, model.summary.weightedPrecision
+// or model.summary.roc which returns a DataFrame
+message FetchAttr {
+ // (Required) reference to the object on the server side or
+ // the intermediate attribute of the model. eg, "model.summary"
+ ObjectRef obj_ref = 1;
+ // (Required) the method name. Eg, "coefficients" of the model
+ // and "weightedPrecision" of "model.summary"
+ string method = 2;
+ // (Optional) the parameters of the method
+ repeated Args args = 3;
+
+ message Args {
Review Comment:
I don't think it's necessary to support named arguments, given that we're leveraging reflection to do the "invoking"
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -97,13 +98,60 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
+ oneof ml_type {
+ Transform transform = 1;
+ FetchAttr fetch_attr = 2;
+ }
+ // Relation to represent transform(input) of the operator
+ // which could be a cached model or a new transformer
+ message Transform {
+ oneof operator {
+ // Object reference
+ ObjectRef obj_ref = 1;
+ // Could be an ML transformer like VectorAssembler
+ MlOperator transformer = 2;
+ }
+ // the input dataframe
+ Relation input = 3;
+ // the operator specific parameters
+ MlParams params = 4;
+ }
+}
+
+// Message for fetching attribute from object on the server side.
+// FetchAttr can be represented as a Relation or a ML command
Review Comment:
Hmm, Some attributes return the literal results, while the other return the DataFrame.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ protected lazy val instance = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(instance, lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+}
+
+private class ModelAttributeHelper(
+ sessionHolder: SessionHolder,
+ objIdentifier: String,
+ method: Option[String],
+ argValues: Array[Object] = Array.empty,
+ argClasses: Array[Class[_]] = Array.empty)
+ extends AttributeHelper(sessionHolder, objIdentifier, method, argValues, argClasses) {
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val model = instance.asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object AttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): AttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new AttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+// MLHandler is a utility to group all ML operations
+object MLHandler extends Logging {
+ def handleMlCommand(
+ sessionHolder: SessionHolder,
+ mlCommand: proto.MlCommand): proto.MlCommandResult = {
+
+ val mlCache = sessionHolder.mlCache
+
+ mlCommand.getCommandCase match {
+ case proto.MlCommand.CommandCase.FIT =>
+ val fitCmd = mlCommand.getFit
+ val estimatorProto = fitCmd.getEstimator
+ assert(estimatorProto.getType == proto.MlOperator.OperatorType.ESTIMATOR)
+
+ val dataset = MLUtils.parseRelationProto(fitCmd.getDataset, sessionHolder)
+ val estimator = MLUtils.getEstimator(estimatorProto, Some(fitCmd.getParams))
+ val model = estimator.fit(dataset).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id)))
+ .build()
+
+ case proto.MlCommand.CommandCase.FETCH_ATTR =>
+ val args = mlCommand.getFetchAttr.getArgsList.asScala.toArray
+ val helper = AttributeHelper(
+ sessionHolder,
+ mlCommand.getFetchAttr.getObjRef.getId,
+ Option(mlCommand.getFetchAttr.getMethod),
+ args)
+ val attrResult = helper.getAttribute
+ attrResult match {
+ case s: Summary =>
+ val id = mlCache.register(s)
+ proto.MlCommandResult.newBuilder().setSummary(id).build()
+ case _ =>
+ val param = Serializer.serializeParam(attrResult)
+ proto.MlCommandResult.newBuilder().setParam(param).build()
+ }
+
+ case proto.MlCommand.CommandCase.DELETE =>
+ val modelId = mlCommand.getDelete.getObjRef.getId
+ var result = false
+ if (!modelId.contains(".")) {
+ mlCache.remove(modelId)
+ result = true
+ }
+ proto.MlCommandResult
+ .newBuilder()
+ .setParam(
+ proto.Param
+ .newBuilder()
+ .setLiteral(LiteralValueProtoConverter.toLiteralProto(result))
+ .build())
+ .build()
+
+ case proto.MlCommand.CommandCase.WRITE =>
+ mlCommand.getWrite.getTypeCase match {
+ case proto.MlCommand.Writer.TypeCase.OBJ_REF => // save a model
+ val modelId = mlCommand.getWrite.getObjRef.getId
+ val model = mlCache.get(modelId).asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, mlCommand.getWrite.getParams)
+
+ copiedModel match {
+ case m: MLWritable => MLUtils.write(m, mlCommand.getWrite)
+ case other => throw MlUnsupportedException(s"$other is not writable")
+ }
+
+ // save an estimator/evaluator/transformer
+ case proto.MlCommand.Writer.TypeCase.OPERATOR =>
+ val writer = mlCommand.getWrite
+ if (writer.getOperator.getType == proto.MlOperator.OperatorType.ESTIMATOR) {
+ val estimator = MLUtils.getEstimator(writer.getOperator, Some(writer.getParams))
+ estimator match {
+ case m: MLWritable => MLUtils.write(m, mlCommand.getWrite)
+ case other => throw MlUnsupportedException(s"Estimator $other is not writable")
+ }
+ } else {
+ throw MlUnsupportedException(s"${writer.getOperator.getName} not supported")
+ }
+
+ case other => throw MlUnsupportedException(s"$other not supported")
+ }
+ proto.MlCommandResult.newBuilder().build()
+
+ case proto.MlCommand.CommandCase.READ =>
+ val operator = mlCommand.getRead.getOperator
+ val name = operator.getName
+ val path = mlCommand.getRead.getPath
+
+ if (operator.getType == proto.MlOperator.OperatorType.MODEL) {
+ val model = MLUtils.load(name, path).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id))
+ .setUid(model.uid)
+ .setParams(Serializer.serializeParams(model)))
+ .build()
+
+ } else if (operator.getType == proto.MlOperator.OperatorType.ESTIMATOR) {
+ val estimator = MLUtils.load(name, path).asInstanceOf[Estimator[_]]
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setName(name)
+ .setUid(estimator.uid)
+ .setParams(Serializer.serializeParams(estimator)))
+ .build()
+ } else {
+ throw MlUnsupportedException(s"${operator.getType} not supported")
+ }
+
+ case other => throw MlUnsupportedException(s"$other not supported")
+ }
+ }
+
+ def transformMLRelation(relation: proto.MlRelation, sessionHolder: SessionHolder): DataFrame = {
Review Comment:
Hmm, This function indeed returns the Dataframe, but the caller will return its logicalPlan from https://github.com/apache/spark/pull/48791/files#diff-db5ed413eb10608faef1340838a236cbbac5ca71ad5193a61cc953c66a01c12fR234
``` scala
case proto.Command.CommandTypeCase.ML_COMMAND =>
handleMlCommand(command.getMlCommand, responseObserver)
```
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLUtils.scala:
##########
@@ -0,0 +1,293 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import java.util.ServiceLoader
+
+import scala.collection.immutable.HashSet
+import scala.jdk.CollectionConverters.{IterableHasAsScala, MapHasAsScala}
+
+import org.apache.commons.lang3.reflect.MethodUtils.{invokeMethod, invokeStaticMethod}
+
+import org.apache.spark.connect.proto
+import org.apache.spark.ml.{Estimator, Transformer}
+import org.apache.spark.ml.linalg.{Matrices, Matrix, Vector, Vectors}
+import org.apache.spark.ml.param.Params
+import org.apache.spark.ml.util.MLWritable
+import org.apache.spark.sql.{DataFrame, Dataset}
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.planner.SparkConnectPlanner
+import org.apache.spark.sql.connect.service.SessionHolder
+import org.apache.spark.util.Utils
+
+private[ml] object MLUtils {
+
+ private lazy val estimators: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Estimator[_]], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ private lazy val transformers: Map[String, Class[_]] = {
+ val loader = Utils.getContextOrSparkClassLoader
+ val serviceLoader = ServiceLoader.load(classOf[Transformer], loader)
+ val providers = serviceLoader.asScala.toList
+ providers.map(est => est.getClass.getName -> est.getClass).toMap
+ }
+
+ def deserializeVector(vector: proto.Vector): Vector = {
+ if (vector.hasDense) {
+ val values = vector.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.dense(values)
+ } else {
+ val size = vector.getSparse.getSize
+ val indices = vector.getSparse.getIndexList.asScala.map(_.toInt).toArray
+ val values = vector.getSparse.getValueList.asScala.map(_.toDouble).toArray
+ Vectors.sparse(size, indices, values)
+ }
+ }
+
+ def deserializeMatrix(matrix: proto.Matrix): Matrix = {
+ if (matrix.hasDense) {
+ val values = matrix.getDense.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.dense(matrix.getDense.getNumRows, matrix.getDense.getNumCols, values)
+ } else {
+ val sparse = matrix.getSparse
+ val colPtrs = sparse.getColptrList.asScala.map(_.toInt).toArray
+ val rowIndices = sparse.getRowIndexList.asScala.map(_.toInt).toArray
+ val values = sparse.getValueList.asScala.map(_.toDouble).toArray
+ Matrices.sparse(sparse.getNumRows, sparse.getNumCols, colPtrs, rowIndices, values)
+ }
+ }
+
+ def setInstanceParams(instance: Params, params: proto.MlParams): Unit = {
+ params.getParamsMap.asScala.foreach { case (name, paramProto) =>
+ val p = instance.getParam(name)
+ val value = if (paramProto.hasLiteral) {
+ convertParamValue(
+ p.paramValueClassTag.runtimeClass,
+ LiteralValueProtoConverter.toCatalystValue(paramProto.getLiteral))
+ } else if (paramProto.hasVector) {
+ deserializeVector(paramProto.getVector)
+ } else if (paramProto.hasMatrix) {
+ deserializeMatrix(paramProto.getMatrix)
+ } else {
+ throw MlUnsupportedException(s"Unsupported parameter type for ${name}")
+ }
+ instance.set(p, value)
+ }
+ }
+
+ private def convertArray(paramType: Class[_], array: Array[_]): Array[_] = {
+ if (paramType == classOf[Byte]) {
+ array.map(_.asInstanceOf[Byte])
+ } else if (paramType == classOf[Short]) {
+ array.map(_.asInstanceOf[Short])
+ } else if (paramType == classOf[Int]) {
+ array.map(_.asInstanceOf[Int])
+ } else if (paramType == classOf[Long]) {
+ array.map(_.asInstanceOf[Long])
+ } else if (paramType == classOf[Float]) {
+ array.map(_.asInstanceOf[Float])
+ } else if (paramType == classOf[Double]) {
+ array.map(_.asInstanceOf[Double])
+ } else if (paramType == classOf[String]) {
+ array.map(_.asInstanceOf[String])
+ } else {
+ array
+ }
+ }
+
+ private def convertParamValue(paramType: Class[_], value: Any): Any = {
+ // Some cases the param type might be mismatched with the value type.
+ // Because in python side we only have int / float type for numeric params.
+ // e.g.:
+ // param type is Int but client sends a Long type.
+ // param type is Long but client sends a Int type.
+ // param type is Float but client sends a Double type.
+ // param type is Array[Int] but client sends a Array[Long] type.
+ // param type is Array[Float] but client sends a Array[Double] type.
+ // param type is Array[Array[Int]] but client sends a Array[Array[Long]] type.
+ // param type is Array[Array[Float]] but client sends a Array[Array[Double]] type.
+ if (paramType == classOf[Byte]) {
+ value.asInstanceOf[java.lang.Number].byteValue()
+ } else if (paramType == classOf[Short]) {
+ value.asInstanceOf[java.lang.Number].shortValue()
+ } else if (paramType == classOf[Int]) {
+ value.asInstanceOf[java.lang.Number].intValue()
+ } else if (paramType == classOf[Long]) {
+ value.asInstanceOf[java.lang.Number].longValue()
+ } else if (paramType == classOf[Float]) {
+ value.asInstanceOf[java.lang.Number].floatValue()
+ } else if (paramType == classOf[Double]) {
+ value.asInstanceOf[java.lang.Number].doubleValue()
+ } else if (paramType.isArray) {
+ val compType = paramType.getComponentType
+ val array = value.asInstanceOf[Array[_]].map { e =>
+ convertParamValue(compType, e)
+ }
+ convertArray(compType, array)
+ } else {
+ value
+ }
+ }
+
+ def parseRelationProto(relation: proto.Relation, sessionHolder: SessionHolder): DataFrame = {
+ val planner = new SparkConnectPlanner(sessionHolder)
+ val plan = planner.transformRelation(relation)
+ Dataset.ofRows(sessionHolder.session, plan)
+ }
+
+ /**
+ * Get the Estimator instance according to the proto information
+ *
+ * @param operator
+ * MlOperator information
+ * @param params
+ * The optional parameters of the estimator
+ * @return
+ * the estimator
+ */
+ def getEstimator(operator: proto.MlOperator, params: Option[proto.MlParams]): Estimator[_] = {
+ // TODO support plugin
+ // Get the estimator according to the operator name
+ val name = operator.getName
+ if (estimators.isEmpty || !estimators.contains(name)) {
+ throw MlUnsupportedException(s"Unsupported Estimator, found ${name}")
+ }
+ val uid = operator.getUid
+ val estimator: Estimator[_] = estimators(name)
Review Comment:
Typically, we just construct the estimator once and use it train a dataset and return a model. Then we don't need it anymore.
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
+ oneof command {
+ Fit fit = 1;
+ FetchAttr fetch_attr = 2;
+ Delete delete = 3;
+ Writer write = 4;
+ Reader read = 5;
Review Comment:
Sound good, Thx
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
Review Comment:
We could put the ML specific commands into the existing Command, just like StreamingQueryCommand which groups corresponding commands together, I'd prefer to group ML-specific commands together.
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
+ oneof command {
+ Fit fit = 1;
+ FetchAttr fetch_attr = 2;
+ Delete delete = 3;
+ Writer write = 4;
Review Comment:
sounds good. thx
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -97,13 +98,60 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
+ oneof ml_type {
+ Transform transform = 1;
+ FetchAttr fetch_attr = 2;
+ }
+ // Relation to represent transform(input) of the operator
+ // which could be a cached model or a new transformer
+ message Transform {
+ oneof operator {
+ // Object reference
+ ObjectRef obj_ref = 1;
+ // Could be an ML transformer like VectorAssembler
+ MlOperator transformer = 2;
+ }
+ // the input dataframe
+ Relation input = 3;
+ // the operator specific parameters
+ MlParams params = 4;
+ }
+}
+
+// Message for fetching attribute from object on the server side.
+// FetchAttr can be represented as a Relation or a ML command
+// Eg, model.coefficients, model.summary.weightedPrecision
+// or model.summary.roc which returns a DataFrame
+message FetchAttr {
Review Comment:
Yeah, it is just passing the arguments of the attribute/properties/method in python to the server side to get the final result. But there're two different scenarios,
1. Get the attribute from a cached model.
For this case, it's quite simple, obj_ref is the id of the cached model, the method is the name of attribute/property/method, and the args are the arguments of the method.
2. Get the attribute from an intermediate object.
For this case, let's take `model.summary` as an example, model.summary returns a Summary object from a cached model on the server side, for example, LogisticRegressionTrainingSummary. The python side also has a corresponding `LogisticRegressionTrainingSummary` class which also has the corresponding attributes. So after having the object of LogisticRegressionTrainingSummary on the python side, we could access the attributes of LogisticRegressionTrainingSummary on the python side. So what this PR does for this case?
Firstly, when users call `model.summary` which will not trigger any GRPC call, instead, python client stores "model.summary" into the LogisticRegressionTrainingSummary.
Secondly. when users call `model.summary.weightedPrecision`, python client side creates Fetch proto by setting below fields
```
obj_ref="model.summary"
method="weightedPrecision"
args="arguments of weightedPrecision"
```
The server side will do after receiving Fetch command
find the model instance from cache according to the value of obj_ref. and then calling `summary` to get an object of LogisticRegressionTrainingSummary and call weightedPrecision on the object and return the final result.
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
+ private val modelId = methodChains.head
+
+ protected lazy val instance = sessionHolder.mlCache.get(modelId)
+ private lazy val methods = methodChains.slice(1, methodChains.length)
+
+ def getAttribute: Any = {
+ assert(methods.length >= 1)
+ if (argValues.length == 0) {
+ methods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ } else {
+ val lastMethod = methods.last
+ if (methods.length == 1) {
+ MLUtils.invokeMethodAllowed(instance, lastMethod, argValues, argClasses)
+ } else {
+ val prevMethods = methods.slice(0, methods.length - 1)
+ val finalObj = prevMethods.foldLeft(instance) { (obj, attribute) =>
+ MLUtils.invokeMethodAllowed(obj, attribute)
+ }
+ MLUtils.invokeMethodAllowed(finalObj, lastMethod, argValues, argClasses)
+ }
+ }
+ }
+}
+
+private class ModelAttributeHelper(
+ sessionHolder: SessionHolder,
+ objIdentifier: String,
+ method: Option[String],
+ argValues: Array[Object] = Array.empty,
+ argClasses: Array[Class[_]] = Array.empty)
+ extends AttributeHelper(sessionHolder, objIdentifier, method, argValues, argClasses) {
+
+ def transform(relation: proto.MlRelation.Transform): DataFrame = {
+ // Create a copied model to avoid concurrently modify model params.
+ val model = instance.asInstanceOf[Model[_]]
+ val copiedModel = model.copy(ParamMap.empty).asInstanceOf[Model[_]]
+ MLUtils.setInstanceParams(copiedModel, relation.getParams)
+ val inputDF = MLUtils.parseRelationProto(relation.getInput, sessionHolder)
+ copiedModel.transform(inputDF)
+ }
+}
+
+private object AttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): AttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new AttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+private object ModelAttributeHelper {
+ def apply(
+ sessionHolder: SessionHolder,
+ modelId: String,
+ method: Option[String] = None,
+ args: Array[proto.FetchAttr.Args] = Array.empty): ModelAttributeHelper = {
+ val tmp = deserializeMethodArguments(args, sessionHolder)
+ val argValues = tmp.map(_._1)
+ val argClasses = tmp.map(_._2)
+ new ModelAttributeHelper(sessionHolder, modelId, method, argValues, argClasses)
+ }
+}
+
+// MLHandler is a utility to group all ML operations
+object MLHandler extends Logging {
+ def handleMlCommand(
+ sessionHolder: SessionHolder,
+ mlCommand: proto.MlCommand): proto.MlCommandResult = {
+
+ val mlCache = sessionHolder.mlCache
+
+ mlCommand.getCommandCase match {
+ case proto.MlCommand.CommandCase.FIT =>
+ val fitCmd = mlCommand.getFit
+ val estimatorProto = fitCmd.getEstimator
+ assert(estimatorProto.getType == proto.MlOperator.OperatorType.ESTIMATOR)
+
+ val dataset = MLUtils.parseRelationProto(fitCmd.getDataset, sessionHolder)
+ val estimator = MLUtils.getEstimator(estimatorProto, Some(fitCmd.getParams))
+ val model = estimator.fit(dataset).asInstanceOf[Model[_]]
+ val id = mlCache.register(model)
+ proto.MlCommandResult
+ .newBuilder()
+ .setOperatorInfo(
+ proto.MlCommandResult.MlOperatorInfo
+ .newBuilder()
+ .setObjRef(proto.ObjectRef.newBuilder().setId(id)))
+ .build()
+
+ case proto.MlCommand.CommandCase.FETCH_ATTR =>
+ val args = mlCommand.getFetchAttr.getArgsList.asScala.toArray
+ val helper = AttributeHelper(
+ sessionHolder,
+ mlCommand.getFetchAttr.getObjRef.getId,
+ Option(mlCommand.getFetchAttr.getMethod),
+ args)
+ val attrResult = helper.getAttribute
+ attrResult match {
+ case s: Summary =>
+ val id = mlCache.register(s)
+ proto.MlCommandResult.newBuilder().setSummary(id).build()
+ case _ =>
+ val param = Serializer.serializeParam(attrResult)
+ proto.MlCommandResult.newBuilder().setParam(param).build()
+ }
+
+ case proto.MlCommand.CommandCase.DELETE =>
+ val modelId = mlCommand.getDelete.getObjRef.getId
+ var result = false
+ if (!modelId.contains(".")) {
Review Comment:
For example, the "modeld" here could be a model reference like AABBCC, or attribute of model, for example "AABBCC.summary", but the server side only caches the model, it didn't cache the summary. so we need to exclude this scenary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876956827
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
Review Comment:
NVM - I am dense :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876963110
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -97,13 +98,60 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
+ oneof ml_type {
+ Transform transform = 1;
+ FetchAttr fetch_attr = 2;
+ }
+ // Relation to represent transform(input) of the operator
+ // which could be a cached model or a new transformer
+ message Transform {
+ oneof operator {
+ // Object reference
+ ObjectRef obj_ref = 1;
+ // Could be an ML transformer like VectorAssembler
+ MlOperator transformer = 2;
+ }
+ // the input dataframe
+ Relation input = 3;
+ // the operator specific parameters
+ MlParams params = 4;
+ }
+}
+
+// Message for fetching attribute from object on the server side.
+// FetchAttr can be represented as a Relation or a ML command
Review Comment:
Why both a command and a relation? It is a read operation, so I don't understand why it should be a command.
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -97,13 +98,60 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
+ oneof ml_type {
+ Transform transform = 1;
+ FetchAttr fetch_attr = 2;
+ }
+ // Relation to represent transform(input) of the operator
+ // which could be a cached model or a new transformer
+ message Transform {
+ oneof operator {
+ // Object reference
+ ObjectRef obj_ref = 1;
+ // Could be an ML transformer like VectorAssembler
+ MlOperator transformer = 2;
+ }
+ // the input dataframe
+ Relation input = 3;
+ // the operator specific parameters
+ MlParams params = 4;
+ }
+}
+
+// Message for fetching attribute from object on the server side.
+// FetchAttr can be represented as a Relation or a ML command
Review Comment:
Why both a command and a relation? It is a read operation, so I don't understand why it could be a command.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876860426
##########
sql/connect/common/src/main/protobuf/spark/connect/ml_common.proto:
##########
@@ -0,0 +1,107 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// MlParams stores param settings for ML Estimator / Transformer / Evaluator
+message MlParams {
+ // User-supplied params
+ map<string, Param> params = 1;
+}
+
+// Represents the parameter type of the ML instance, or the returned value
+// of the attribute
+message Param {
+ oneof param_type {
+ Expression.Literal literal = 1;
+ Vector vector = 2;
+ Matrix matrix = 3;
+ }
+}
+
+// MLOperator represents the ML operators like (Estimator, Transformer or Evaluator)
+message MlOperator {
+ // The qualified name of the ML operator.
+ string name = 1;
+ // Unique id of the ML operator
+ string uid = 2;
Review Comment:
Is this UID the same as the one used in ObjectRef?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1876983288
##########
sql/connect/common/src/main/protobuf/spark/connect/ml.proto:
##########
@@ -0,0 +1,106 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+syntax = 'proto3';
+
+package spark.connect;
+
+import "spark/connect/expressions.proto";
+import "spark/connect/relations.proto";
+import "spark/connect/ml_common.proto";
+
+option java_multiple_files = true;
+option java_package = "org.apache.spark.connect.proto";
+
+// Command for ML
+message MlCommand {
Review Comment:
One more question here. Why is indirection needed? Why not add Fit/Fetch/Delete/Writer/Reader to the oneof in Command?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1883376090
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -98,13 +99,61 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
+ oneof ml_type {
+ Transform transform = 1;
+ Fetch fetch = 2;
+ }
+ // Relation to represent transform(input) of the operator
+ // which could be a cached model or a new transformer
+ message Transform {
+ oneof operator {
+ // Object reference
+ ObjectRef obj_ref = 1;
+ // Could be an ML transformer like VectorAssembler
+ MlOperator transformer = 2;
+ }
+ // the input dataframe
+ Relation input = 3;
+ // the operator specific parameters
+ MlParams params = 4;
+ }
+}
+
+// Message for fetching attribute from object on the server side.
+// Fetch can be represented as a Relation or a ML command
+// Command: model.coefficients, model.summary.weightedPrecision which
+// returns the final literal result
+// Relation: model.summary.roc which returns a DataFrame (Relation
+message Fetch {
+ // (Required) reference to the object on the server side or
+ // the intermediate attribute of the model. eg, "model.summary"
+ ObjectRef obj_ref = 1;
+ // (Required) the method name. Eg, "coefficients" of the model
+ // and "weightedPrecision" of the "model.summary"
+ string method = 2;
+ // (Optional) the arguments of the method
+ repeated Args args = 3;
Review Comment:
I just refactor Fetch a little bit, could you review the latest commit. Thx very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "grundprinzip (via GitHub)" <gi...@apache.org>.
grundprinzip commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1882378356
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
Review Comment:
I think what @hvanhovell means is that we should just use a repeated field in the proto so that we don't have to hard code the splitting and it's more obvious what the usage of the field is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1883115637
##########
sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ml/MLHandler.scala:
##########
@@ -0,0 +1,276 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connect.ml
+
+import scala.jdk.CollectionConverters.CollectionHasAsScala
+
+import org.apache.spark.connect.proto
+import org.apache.spark.internal.Logging
+import org.apache.spark.ml.{Estimator, Model}
+import org.apache.spark.ml.param.ParamMap
+import org.apache.spark.ml.util.{MLWritable, Summary}
+import org.apache.spark.sql.DataFrame
+import org.apache.spark.sql.connect.common.LiteralValueProtoConverter
+import org.apache.spark.sql.connect.ml.Serializer.deserializeMethodArguments
+import org.apache.spark.sql.connect.service.SessionHolder
+
+/**
+ * Helper function to get the attribute from an object by reflection
+ */
+private class AttributeHelper(
+ val sessionHolder: SessionHolder,
+ val objIdentifier: String,
+ val method: Option[String],
+ val argValues: Array[Object] = Array.empty,
+ val argClasses: Array[Class[_]] = Array.empty) {
+
+ private val methodChain = method.map(n => s"$objIdentifier.$n").getOrElse(objIdentifier)
+ private val methodChains = methodChain.split("\\.")
Review Comment:
Yes that is what I mean. I could have been clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "hvanhovell (via GitHub)" <gi...@apache.org>.
hvanhovell commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1883115341
##########
sql/connect/common/src/main/protobuf/spark/connect/relations.proto:
##########
@@ -98,13 +99,61 @@ message Relation {
// Catalog API (experimental / unstable)
Catalog catalog = 200;
+ // ML relation
+ MlRelation ml_relation = 300;
+
// This field is used to mark extensions to the protocol. When plugins generate arbitrary
// relations they can add them here. During the planning the correct resolution is done.
google.protobuf.Any extension = 998;
Unknown unknown = 999;
}
}
+// Relation to represent ML world
+message MlRelation {
+ oneof ml_type {
+ Transform transform = 1;
+ Fetch fetch = 2;
+ }
+ // Relation to represent transform(input) of the operator
+ // which could be a cached model or a new transformer
+ message Transform {
+ oneof operator {
+ // Object reference
+ ObjectRef obj_ref = 1;
+ // Could be an ML transformer like VectorAssembler
+ MlOperator transformer = 2;
+ }
+ // the input dataframe
+ Relation input = 3;
+ // the operator specific parameters
+ MlParams params = 4;
+ }
+}
+
+// Message for fetching attribute from object on the server side.
+// Fetch can be represented as a Relation or a ML command
+// Command: model.coefficients, model.summary.weightedPrecision which
+// returns the final literal result
+// Relation: model.summary.roc which returns a DataFrame (Relation
+message Fetch {
+ // (Required) reference to the object on the server side or
+ // the intermediate attribute of the model. eg, "model.summary"
+ ObjectRef obj_ref = 1;
+ // (Required) the method name. Eg, "coefficients" of the model
+ // and "weightedPrecision" of the "model.summary"
+ string method = 2;
+ // (Optional) the arguments of the method
+ repeated Args args = 3;
Review Comment:
Can you tell me which methods take arguments? I would prefer if those actually have dedicated RPCs, or alternatively we expose all of this through specialized expressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2507092791
Hi @grundprinzip, I just updated this PR by supporting sparse vector/matrix and supporting the evaluate a dataset in a model. So for now, All attributes in LogisticRegressionModel have been supported. Please help review it again. Thx
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "zhengruifeng (via GitHub)" <gi...@apache.org>.
zhengruifeng commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2589673471
thanks, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on PR #48791:
URL: https://github.com/apache/spark/pull/48791#issuecomment-2487461014
Hi @WeichenXu123, @grundprinzip, @mengxr, Could you help review this PR. the CI got passed. Thx very much.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49907][ML][CONNECT] Support spark.ml on Connect [spark]
Posted by "wbo4958 (via GitHub)" <gi...@apache.org>.
wbo4958 commented on code in PR #48791:
URL: https://github.com/apache/spark/pull/48791#discussion_r1846144511
##########
python/pyspark/ml/remote/util.py:
##########
@@ -0,0 +1,238 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import functools
+import os
+from typing import Any, cast, TypeVar, Callable, TYPE_CHECKING, Type
+
+import pyspark.sql.connect.proto as pb2
+from pyspark.ml.remote.serialize import serialize_ml_params, serialize, deserialize
+from pyspark.sql import is_remote
+from pyspark.sql.connect.dataframe import DataFrame as RemoteDataFrame
+
+if TYPE_CHECKING:
+ from pyspark.ml.wrapper import JavaWrapper, JavaEstimator
+ from pyspark.ml.util import JavaMLReadable, JavaMLWritable
+
+FuncT = TypeVar("FuncT", bound=Callable[..., Any])
+
+
+def try_remote_intermediate_result(f: FuncT) -> FuncT:
+ """Mark the function/property that returns the intermediate result of the remote call.
+ Eg, model.summary"""
+
+ @functools.wraps(f)
+ def wrapped(self: "JavaWrapper") -> Any:
+ if is_remote() and "PYSPARK_NO_NAMESPACE_SHARE" not in os.environ:
+ return f"{self._java_obj}.{f.__name__}"
Review Comment:
This is for the attributes that return a class instead of final values like vector/literal/matrix. Eg, lr.summary which returns LogisticRegressionTrainingSummary. we just put the "cached_model_id.summary" into LogisticRegressionTrainingSummary without triggering any RPC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org