You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2025/03/31 23:44:00 UTC
[jira] [Resolved] (SPARK-51537) Failed to run third-party Spark ML library on Spark Connect
[ https://issues.apache.org/jira/browse/SPARK-51537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-51537.
----------------------------------
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 50334
[https://github.com/apache/spark/pull/50334]
> Failed to run third-party Spark ML library on Spark Connect
> ------------------------------------------------------------
>
> Key: SPARK-51537
> URL: https://issues.apache.org/jira/browse/SPARK-51537
> Project: Spark
> Issue Type: Bug
> Components: Connect, ML
> Affects Versions: 4.0.0
> Reporter: Bobby Wang
> Assignee: Bobby Wang
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> I've encountered an issue where the third-party Spark ML library may not run
> on Spark Connect. This problem occurs when specifying the
> third-party ML jar using the *--jars* configuration while creating a connect server
> based on a Spark standalone cluster.
>
> The exception thrown is a ClassCastException:
>
> _Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.rdd.MapPartitionsRDD.f of type scala.Function3 in instance of org.apache.spark.rdd.MapPartitionsRDD_
> _at java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2096)_
>
> However, if I place the ML jar into the *$SPARK_HOME/jars* directory and restart both the Spark standalone cluster and the Spark Connect server, it runs without any exceptions.
>
> Alternatively, adding *spark.addArtifacts("target/com.example.ml-1.0-SNAPSHOT.jar")* directly in the python code also resolves the issue.
>
> I have made a minimum project which can repro this issue, more details could be found at [https://github.com/wbo4958/ConnectMLIssue]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org