You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by HanPan <pa...@thinkingdata.cn> on 2016/03/21 04:31:51 UTC

MLPC model can not be saved

 

Hi Guys,

 

         I built a ML pipeline that includes multilayer perceptron
classifier, I got the following error message when I tried to save the
pipeline model. It seems like MLPC model can not be saved which means I have
no ways to save the trained model. Is there any way to save the model that I
can use it for future prediction.

 

         Exception in thread "main" java.lang.UnsupportedOperationException:
Pipeline write will fail on this Pipeline because it contains a stage which
does not implement Writable. Non-Writable stage: mlpc_2d8b74f6da60 of type
class
org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel

         at
org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$validateStages$1.apply
(Pipeline.scala:218)

         at
org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$validateStages$1.apply
(Pipeline.scala:215)

         at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala
:33)

         at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)

         at
org.apache.spark.ml.Pipeline$SharedReadWrite$.validateStages(Pipeline.scala:
215)

         at
org.apache.spark.ml.PipelineModel$PipelineModelWriter.<init>(Pipeline.scala:
325)

         at org.apache.spark.ml.PipelineModel.write(Pipeline.scala:309)

         at
org.apache.spark.ml.util.MLWritable$class.save(ReadWrite.scala:130)

         at org.apache.spark.ml.PipelineModel.save(Pipeline.scala:280)

         at
cn.thinkingdata.nlp.spamclassifier.FFNNSpamClassifierPipeLine.main(FFNNSpamC
lassifierPipeLine.java:76)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62
)

         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

         at java.lang.reflect.Method.invoke(Method.java:497)

         at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$ru
nMain(SparkSubmit.scala:731)

         at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)

         at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)

         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

Thanks

Pan

答复: MLPC model can not be saved

Posted by HanPan <pa...@thinkingdata.cn>.
Hi Alexander,

 

         Thanks for your reply. The pull request shows that
MultilayerPerceptronClassifier implement default params writable interface.
I will try that.

 

Thanks

Pan

 

发件人: Ulanov, Alexander [mailto:alexander.ulanov@hpe.com] 
发送时间: 2016年3月22日 1:38
收件人: HanPan; dev@spark.apache.org
主题: RE: MLPC model can not be saved

 

Hi Pan,

 

There is a pull request that is supposed to fix the issue:

https://github.com/apache/spark/pull/9854

 

There is a workaround for saving/loading a model (however I am not sure if
it will work for the pipeline): 

sc.parallelize(Seq(model), 1).saveAsObjectFile("path")

val sameModel = sc.objectFile[YourCLASS]("path").first()

 

 

Best regards, Alexander

 

From: HanPan [mailto:panda@thinkingdata.cn] 
Sent: Sunday, March 20, 2016 8:32 PM
To: dev@spark.apache.org <ma...@spark.apache.org> 
Cc: panda@thinkingdata.cn <ma...@thinkingdata.cn> 
Subject: MLPC model can not be saved

 

 

Hi Guys,

 

         I built a ML pipeline that includes multilayer perceptron
classifier, I got the following error message when I tried to save the
pipeline model. It seems like MLPC model can not be saved which means I have
no ways to save the trained model. Is there any way to save the model that I
can use it for future prediction.

 

         Exception in thread "main" java.lang.UnsupportedOperationException:
Pipeline write will fail on this Pipeline because it contains a stage which
does not implement Writable. Non-Writable stage: mlpc_2d8b74f6da60 of type
class
org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel

         at
org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$validateStages$1.apply
(Pipeline.scala:218)

         at
org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$validateStages$1.apply
(Pipeline.scala:215)

         at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala
:33)

         at
scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)

         at
org.apache.spark.ml.Pipeline$SharedReadWrite$.validateStages(Pipeline.scala:
215)

         at
org.apache.spark.ml.PipelineModel$PipelineModelWriter.<init>(Pipeline.scala:
325)

         at org.apache.spark.ml.PipelineModel.write(Pipeline.scala:309)

         at
org.apache.spark.ml.util.MLWritable$class.save(ReadWrite.scala:130)

         at org.apache.spark.ml.PipelineModel.save(Pipeline.scala:280)

         at
cn.thinkingdata.nlp.spamclassifier.FFNNSpamClassifierPipeLine.main(FFNNSpamC
lassifierPipeLine.java:76)

         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

         at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62
)

         at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)

         at java.lang.reflect.Method.invoke(Method.java:497)

         at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$ru
nMain(SparkSubmit.scala:731)

         at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)

         at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)

         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)

         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

 

Thanks

Pan

RE: MLPC model can not be saved

Posted by "Ulanov, Alexander" <al...@hpe.com>.
Hi Pan,

There is a pull request that is supposed to fix the issue:
https://github.com/apache/spark/pull/9854

There is a workaround for saving/loading a model (however I am not sure if it will work for the pipeline):
sc.parallelize(Seq(model), 1).saveAsObjectFile("path")
val sameModel = sc.objectFile[YourCLASS]("path").first()


Best regards, Alexander

From: HanPan [mailto:panda@thinkingdata.cn]
Sent: Sunday, March 20, 2016 8:32 PM
To: dev@spark.apache.org
Cc: panda@thinkingdata.cn
Subject: MLPC model can not be saved


Hi Guys,

         I built a ML pipeline that includes multilayer perceptron classifier, I got the following error message when I tried to save the pipeline model. It seems like MLPC model can not be saved which means I have no ways to save the trained model. Is there any way to save the model that I can use it for future prediction.

         Exception in thread "main" java.lang.UnsupportedOperationException: Pipeline write will fail on this Pipeline because it contains a stage which does not implement Writable. Non-Writable stage: mlpc_2d8b74f6da60 of type class org.apache.spark.ml.classification.MultilayerPerceptronClassificationModel
         at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$validateStages$1.apply(Pipeline.scala:218)
         at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun$validateStages$1.apply(Pipeline.scala:215)
         at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
         at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:108)
         at org.apache.spark.ml.Pipeline$SharedReadWrite$.validateStages(Pipeline.scala:215)
         at org.apache.spark.ml.PipelineModel$PipelineModelWriter.<init>(Pipeline.scala:325)
         at org.apache.spark.ml.PipelineModel.write(Pipeline.scala:309)
         at org.apache.spark.ml.util.MLWritable$class.save(ReadWrite.scala:130)
         at org.apache.spark.ml.PipelineModel.save(Pipeline.scala:280)
         at cn.thinkingdata.nlp.spamclassifier.FFNNSpamClassifierPipeLine.main(FFNNSpamClassifierPipeLine.java:76)
         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
         at java.lang.reflect.Method.invoke(Method.java:497)
         at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
         at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
         at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Thanks
Pan