Learnt lessons to share

[amir bahmanyari <am...@yahoo.com>]

Hi Colleagues,I have been having issues running my Beam / Kafka app for more than a week.I kept getting intermittent class not found exceptions left and right.Till I accidentally changed the name of one of the inner static classes.When I rebuilt, repackaged & redeployed to run, I noticed that Flink complains about that class name BUT with its old name:Caused by: java.lang.ClassNotFoundException: bench.flinkspark.flink.ReadFromKafka2$CompLRRecLengthFn
CompLRRecLengthFn was the old name which I had changed to CompLRRecFn (no Length).
So, I went back, restarted Flink JobManager & my cluster TaskManagers.That "class not found" issue went away. Therefore, it seems like we must recycle the Flink servers , but I am not sure how often!!!Any suggestions pls?
I am still getting an exception regarding that inner static class, but at least its being referred to by its true name "CompLRRecFn".
 java.lang.IllegalArgumentException: unable to deserialize bench.flinkspark.flink.ReadFromKafka2$CompLRRecFn@62dae245        at org.apache.beam.sdk.util.SerializableUtils.deserializeFromByteArray(SerializableUtils.java:78)        at org.apache.beam.sdk.util.SerializableUtils.clone(SerializableUtils.java:93)        at org.apache.beam.sdk.transforms.ParDo$Bound.<init>(ParDo.java:722)

I commented out the invocation to that inner static class.I get p.run() getting to a RUNNING state, waiting for incoming data for a second, and quickly change state to FINISHED.

05/10/2016 15:44:52     Job execution switched to status RUNNING.05/10/2016 15:44:52     Source: Custom File source -> Flat Map -> Timestamps/Watermarks -> Flat Map -> Flat Map -> Flat Map -> Flat Map(1/1) switched to SCHEDULED05/10/2016 15:44:52     Source: Custom File source -> Flat Map -> Timestamps/Watermarks -> Flat Map -> Flat Map -> Flat Map -> Flat Map(1/1) switched to DEPLOYING05/10/2016 15:44:52     Source: Custom File source -> Flat Map -> Timestamps/Watermarks -> Flat Map -> Flat Map -> Flat Map -> Flat Map(1/1) switched to RUNNING05/10/2016 15:44:52     Source: Custom File source -> Flat Map -> Timestamps/Watermarks -> Flat Map -> Flat Map -> Flat Map -> Flat Map(1/1) switched to FINISHED05/10/2016 15:44:52     Job execution switched to status FINISHED.



isn't p.run() thread supposed to stay on for receiving data as a consumer of kafka?
Any idea why it switched to FINISHED immediately pls?
Cheers+ thanks again everyone.