logging pickle files on local run of spark.ml Pipeline model

["Mnisi, Caleb" <cm...@deloitte.co.za.INVALID>]

Good Day

I am having trouble saving a spark.ml Pipeline model to a pickle file, when running locally on my PC.
I've tried a few ways to save the model:

  1.  mlflow.spark.log_model(artifact_path=experiment.artifact_location, spark_model= model, registered_model_name="myModel")
     *   with error that the spark model is multiple files
  2.  pickle.dump(model, file): with error - TypeError: cannot pickle '_thread.RLock' object
  3.  model.save('path'): with Java errors:
     *   at org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
     *   at org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182)
     *   at org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:99) ... 67 more

Your assistance on this would be much appreciated.
Regards,

Caleb Mnisi
Consultant | Deloitte Analytics | Cognitive Advantage
Deloitte & Touche
5th floor, 5 Magwa Crescent, Waterfall City, 2090
M: +27 72 170 8779
cmnisi@deloitte.co.za<ma...@deloitte.co.za> | www2.deloitte.com/za<ma...@deloitte.co.za>

[cid:image001.png@01D95757.1241E560]

Please consider the environment before printing.

*Disclaimer:* This email is subject to important restrictions, qualifications and disclaimers ("the Disclaimer") that must be accessed and read by visiting our website and viewing the webpage at the following address: http://www.deloitte.com/za/disclaimer. The Disclaimer forms part of the content of this email. If you cannot access the Disclaimer, please obtain a copy thereof from us by sending an email to zaitservicedesk@deloitte.co.za. Deloitte refers to a Deloitte member firm, one of its related entities, or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a separate legal entity and a member of DTTL. DTTL does not provide services to clients. Please see www.deloitte.com/about to learn more.

Re: logging pickle files on local run of spark.ml Pipeline model

[Sean Owen <sr...@gmail.com>]

Pickle won't work. But the others should. I think you are specifying an
invalid path in both cases but hard to say without more detail

On Wed, Mar 15, 2023, 9:13 AM Mnisi, Caleb <cm...@deloitte.co.za.invalid>
wrote:

> Good Day
>
>
>
> I am having trouble saving a spark.ml Pipeline model to a pickle file,
> when running locally on my PC.
>
> I’ve tried a few ways to save the model:
>
>    1. mlflow.spark.log_model(artifact_path=experiment.artifact_location,
>    spark_model= model, registered_model_name="myModel")
>       1. with error that the spark model is multiple files
>    2. pickle.dump(model, file): with error - TypeError: cannot pickle
>    '_thread.RLock' object
>    3. model.save(‘path’): with Java errors:
>       1. at
>       org.apache.hadoop.mapred.OutputCommitter.commitJob(OutputCommitter.java:291)
>       2. at
>       org.apache.spark.internal.io.HadoopMapReduceCommitProtocol.commitJob(HadoopMapReduceCommitProtocol.scala:182)
>       3. at
>       org.apache.spark.internal.io.SparkHadoopWriter$.write(SparkHadoopWriter.scala:99)
>       ... 67 more
>
>
>
> Your assistance on this would be much appreciated.
>
> Regards,
>
>
>
> *Caleb Mnisi*
>
> Consultant | Deloitte Analytics | Cognitive Advantage
>
> Deloitte & Touche
>
> 5th floor, 5 Magwa Crescent, Waterfall City, 2090
>
> M: +27 72 170 8779
>
> *cmnisi@deloitte.co.za <cm...@deloitte.co.za>* | www2.deloitte.com/za
> <cm...@deloitte.co.za>
>
>
>
>
>
> Please consider the environment before printing.
>
>
> *Disclaimer:* This email is subject to important restrictions,
> qualifications and disclaimers ("the Disclaimer") that must be accessed and
> read by visiting our website and viewing the webpage at the following
> address: http://www.deloitte.com/za/disclaimer. The Disclaimer forms part
> of the content of this email. If you cannot access the Disclaimer, please
> obtain a copy thereof from us by sending an email to
> zaitservicedesk@deloitte.co.za. Deloitte refers to a Deloitte member
> firm, one of its related entities, or Deloitte Touche Tohmatsu Limited
> (“DTTL”). Each Deloitte member firm is a separate legal entity and a member
> of DTTL. DTTL does not provide services to clients. Please see
> www.deloitte.com/about to learn more.
>