[jira] [Comment Edited] (FLINK-4331) Flink is not able to serialize scala classes / Task Not Serializable

["Pushpendra Jaiswal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-4331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415360#comment-15415360 ] 

Pushpendra Jaiswal edited comment on FLINK-4331 at 8/10/16 2:36 PM:
--------------------------------------------------------------------

Hi [~StephanEwen]
Thanks for the update and sorry for late replying.
This error only arises if I am accessing fields directly (if that's the reference you meant).
If impression.map is replaced by a getter impression.getMap() if doesn't throw an error.
If this behavior is ok, please let me know.
Sure you can close the ticket as I have solved the problem.

~Pushpendra Jaiswal









was (Author: pushpendra.jaiswal90@gmail.com):
Hi [~StephanEwen]
Thanks for the update and sorry for late replying.
This error only arises if I am accessing fields directly (if that's what it reference you meant).
If impression.map is replaced by a getter impression.getMap() if doesn't throw an error.
If this behavior is ok, please let me know.
Sure you can close the ticket as I have solved the problem.

~Pushpendra Jaiswal








> Flink is not able to serialize scala classes / Task Not Serializable
> --------------------------------------------------------------------
>
>                 Key: FLINK-4331
>                 URL: https://issues.apache.org/jira/browse/FLINK-4331
>             Project: Flink
>          Issue Type: Bug
>          Components: DataStream API
>    Affects Versions: 1.1.0
>            Reporter: Pushpendra Jaiswal
>
> I have scala class having 2 fields which are vals but flink is saying it doesn't have setters. Thus task is not serializable.
> I tried setters using var but then it says duplicate setter. vals are public then why it is asking for setters. Flink version 1.1.0
> class Impression(val map: Map[String, String],val keySet:Set[String])
> ==========================================================================
>   val preAggregate = stream
> .filter(impression => {
>     true
> })
>  .map(impression => {
>   val xmap = impression.map
>   val values = valFunction(xmap)
>   new ImpressionRecord(impression, values._1, values._2, values._3)
> })
> class Impression does not contain a setter for field map 19:54:49.995 [main] INFO o.a.f.a.java.typeutils.TypeExtractor - class Impression is not a valid POJO type 19:54:49.997 [main] DEBUG o.a.flink.api.scala.ClosureCleaner$ - accessedFields: Map(class -> Set()) Exception in thread "main" org.apache.flink.api.common.InvalidProgramException: Task not serializable at org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:172) at ) Caused by: java.io.NotSerializableException: org.apache.flink.streaming.api.scala.DataStream at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508) at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431) at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177) at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347) at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:301) at org.apache.flink.api.scala.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:170) ... 18 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)