You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Artemis User <ar...@dtechspace.com> on 2021/01/19 19:29:55 UTC

Persisting Customized Transformer

We are trying to create a customized transformer for a ML pipeline and 
also want to persist the trained pipeline and retrieve it for 
production.  To enable persistency, we will have to implement read/write 
functions.  However, this is not feasible in Scala since the read/write 
methods are private members of the MLModel class.  This problem was 
described in a JIRA ticket https://issues.apache.org/jira/browse/SPARK-17048

Although the ticket suggested some workaround, but only in Java. I was 
wondering if anyone has tried in Scala?  The scala work-around suggested 
in the ticket didn't work for us.  Does anyone know if this issue has 
been resolved in 3.0.1 or the upcoming 3.1?  Any suggestion is highly 
appreciated.

-- ND


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org