You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by erp12 <ed...@gmail.com> on 2018/05/03 15:26:57 UTC
Pickling Keras models for use in UDFs
I would like to create a Spark UDF which returns the a prediction made with a
trained Keras model. Keras models are not typically pickle-able, however I
have used the monkey patch approach to making Keras models pickle-able, as
described here: http://zachmoshe.com/2017/04/03/pickling-keras-models.html
This allows for models to be sent from the PySpark driver to the workers,
however the worker python processes do not have the monkey patched Model
class, and thus cannot properly un-pickle the models. To fix this issue, I
know I must call the monkey patching function (make_keras_picklable()) once
on each worker, however I have been unable to figure out how to do this.
I am curious to hear if anyone has a fix for this issue, or would like to
offer an alternative way to make predictions with a Keras model within a
Spark UDF.
Here is a Stack Overflow question with more details:
https://stackoverflow.com/questions/50007126/pickling-monkey-patched-keras-model-for-use-in-pyspark
Thank you!
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Pickling Keras models for use in UDFs
Posted by erp12 <ed...@gmail.com>.
Great idea! That works perfectly, thank you!
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: Pickling Keras models for use in UDFs
Posted by Khaled Zaouk <kh...@gmail.com>.
Why don't you try to encapsulate your keras model within a wrapper class
(an estimator let's say), and you implement inside this wrapper class the
two functions: __getstate__ and __setstate__
On Thu, May 3, 2018 at 5:27 PM erp12 <ed...@gmail.com> wrote:
> I would like to create a Spark UDF which returns the a prediction made
> with a
> trained Keras model. Keras models are not typically pickle-able, however I
> have used the monkey patch approach to making Keras models pickle-able, as
> described here: http://zachmoshe.com/2017/04/03/pickling-keras-models.html
>
> This allows for models to be sent from the PySpark driver to the workers,
> however the worker python processes do not have the monkey patched Model
> class, and thus cannot properly un-pickle the models. To fix this issue, I
> know I must call the monkey patching function (make_keras_picklable()) once
> on each worker, however I have been unable to figure out how to do this.
>
> I am curious to hear if anyone has a fix for this issue, or would like to
> offer an alternative way to make predictions with a Keras model within a
> Spark UDF.
>
> Here is a Stack Overflow question with more details:
>
> https://stackoverflow.com/questions/50007126/pickling-monkey-patched-keras-model-for-use-in-pyspark
>
> Thank you!
>
>
>
> --
> Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>