Task Serialization Issue while storing model

[Bansari Shah <ba...@gmail.com>]

Hi,
My MLEngine use PAlgorithm for training and predicting model. It requires
to save Model by extending IPersistentModel and IPersistentModelLoader.
I followed this steps and saving below model:

{

def save(id: String, params: AlgorithmParams, sc: SparkContext): Boolean = {
 ldaModel.save(sc, s"/tmp/${id}/ldamodel")
 corpus.saveAsObjectFile(s"/tmp/${id}/ldaCorpus")
 sc.parallelize(Seq(vocab)).saveAsObjectFile(s"/tmp/${id}/ldaVocab")
 true

}

It completes training properly but while saving model it shows below exception:

Log:

...

[INFO] [Engine$] EngineWorkflow.train completed [INFO] [Engine]
engineInstanceId=afcabdee-9f1b-4454-8e80-e20185673769

[ERROR] [InsertIntoHadoopFsRelation] Aborting job.
[ERROR] [DefaultWriterContainer] Job job_201610061606_0000 aborted.
Exception in thread "main" org.apache.spark.SparkException: Job aborted.

...

Caused by: org.apache.spark.SparkException: Job aborted due to stage
failure: Task serialization failed: java.lang.StackOverflowError

Please suggest me how to solve this issue.

Thank you

Regards,
Bansari

Re: Task Serialization Issue while storing model

[Bansari Shah <ba...@gmail.com>]

Thank you Lucas,

My model is DistributedLDA model. I am not sure wether it has huge number
of nested objects. But your suggestion gave me one more insight to think
of. Thank you.

Regards,
Bansari

On Thu, Oct 6, 2016 at 4:43 PM, Lucas Bonatto Miguel <lucasbonatto@gmail.com
> wrote:

> Hello Bansari,
>
> Does your model object have a huge number of nested objects? I'm not sure
> if it is the same case, but I had similar issue when trying to serialize a
> model with around 20 mi nested objects under the same object node. The
> issue was related with kryo not being able to handle this kind of object
> and the solution was spliting and storing it's parts in a Map, you also
> should consider serializing the splitted parts.
>
> Refer to this issue: https://github.com/EsotericSoftware/kryo/issues/103
>
> I hope it can help.
>
> Regards,
> Lucas
>

Re: Task Serialization Issue while storing model

[Lucas Bonatto Miguel <lu...@gmail.com>]

Hello Bansari,

Does your model object have a huge number of nested objects? I'm not sure
if it is the same case, but I had similar issue when trying to serialize a
model with around 20 mi nested objects under the same object node. The
issue was related with kryo not being able to handle this kind of object
and the solution was spliting and storing it's parts in a Map, you also
should consider serializing the splitted parts.

Refer to this issue: https://github.com/EsotericSoftware/kryo/issues/103

I hope it can help.

Regards,
Lucas