You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "Aviem Zur (JIRA)" <ji...@apache.org> on 2017/04/22 07:22:04 UTC
[jira] [Comment Edited] (BEAM-2027) get error sometimes while
running the same code using beam0.6
[ https://issues.apache.org/jira/browse/BEAM-2027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979797#comment-15979797 ]
Aviem Zur edited comment on BEAM-2027 at 4/22/17 7:21 AM:
----------------------------------------------------------
{{local class incompatible}} happens when trying to deserialize a Java class of an older version than the one in your classpath.
The class in question is {{org.apache.beam.runners.spark.coders.CoderHelpers$3}} which is a part of Beam.
In believe in your case the version of Beam in the driver's classpath is different than the one in the executor's classpath.
Spark is serializing an instance of this class in the driver at one version and deserializing it in an executor which expects a different version.
A few questions about your setup:
Did you package your application's jar as a fat jar containing Beam jars?
If not: do you distribute Beam jars to your executors in some other way?
Is this a streaming application? (from your description it didn't sound like you used streaming).
If so, Spark streaming checkpoints certain serialized instances of classes. When changing these in your application and resuming from checkpoint (for example, upgrading Beam version) you can experience failures such as this. The solution for this is deleting the checkpoint and starting fresh.
was (Author: aviemzur):
{{local class incompatible}} happens when trying to deserialize a Java class of an older version than the one in your classpath.
The class in question is {{org.apache.beam.runners.spark.coders.CoderHelpers$3}} which is a part of Beam.
In your case - the version of Beam in the driver's classpath is different than the one in the executor's classpath.
Spark is serializing an instance of this class in the driver at one version and deserializing it in an executor which expects a different version.
A few questions about your setup:
Did you package your application's jar as a fat jar containing Beam jars?
If not: do you distribute Beam jars to your executors in some other way?
Is this a streaming application? (from your description it didn't sound like you used streaming).
If so, Spark streaming checkpoints certain serialized instances of classes. When changing these in your application and resuming from checkpoint (for example, upgrading Beam version) you can experience failures such as this. The solution for this is deleting the checkpoint and starting fresh.
> get error sometimes while running the same code using beam0.6
> -------------------------------------------------------------
>
> Key: BEAM-2027
> URL: https://issues.apache.org/jira/browse/BEAM-2027
> Project: Beam
> Issue Type: Bug
> Components: runner-core, runner-spark
> Environment: spark-1.6.2-bin-hadoop2.6, hadoop-2.6.0, source:hdfs sink:hdfs
> Reporter: liyuntian
> Assignee: Aviem Zur
>
> run a yarn job using beam0.6.0, I get file from hdfs and write record to hdfs, I use spark-1.6.2-bin-hadoop2.6,hadoop-2.6.0. I get error sometime below,
> 17/04/20 21:10:45 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, etl-develop-003): java.io.InvalidClassException: org.apache.beam.runners.spark.coders.CoderHelpers$3; local class incompatible: stream classdesc serialVersionUID = 1334222146820528045, local class serialVersionUID = 5119956493581628999
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at scala.collection.immutable.$colon$colon.readObject(List.scala:362)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1017)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1893)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
> at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:76)
> at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:115)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
> at org.apache.spark.scheduler.Task.run(Task.scala:89)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:227)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Starting task 0.1 in stage 0.0 (TID 2, etl-develop-003, partition 0,PROCESS_LOCAL, 2080 bytes)
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Lost task 0.1 in stage 0.0 (TID 2) on executor etl-develop-003: java.io.InvalidClassException (org.apache.beam.runners.spark.coders.CoderHelpers$3; local class incompatible: stream classdesc serialVersionUID = 1334222146820528045, local class serialVersionUID = 5119956493581628999) [duplicate 1]
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Starting task 0.2 in stage 0.0 (TID 3, etl-develop-003, partition 0,PROCESS_LOCAL, 2080 bytes)
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Lost task 0.2 in stage 0.0 (TID 3) on executor etl-develop-003: java.io.InvalidClassException (org.apache.beam.runners.spark.coders.CoderHelpers$3; local class incompatible: stream classdesc serialVersionUID = 1334222146820528045, local class serialVersionUID = 5119956493581628999) [duplicate 2]
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Starting task 0.3 in stage 0.0 (TID 4, etl-develop-003, partition 0,PROCESS_LOCAL, 2080 bytes)
> 17/04/20 21:10:45 INFO scheduler.TaskSetManager: Lost task 0.3 in stage 0.0 (TID 4) on executor etl-develop-003: java.io.InvalidClassException (org.apache.beam.runners.spark.coders.CoderHelpers$3; local class incompatible: stream classdesc serialVersionUID = 1334222146820528045, local class serialVersionUID = 5119956493581628999) [duplicate 3]
> 17/04/20 21:10:45 ERROR scheduler.TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job
> 17/04/20 21:10:45 INFO cluster.YarnClusterScheduler: Cancelling stage 0
> 17/04/20 21:10:45 INFO cluster.YarnClusterScheduler: Stage 0 was cancelled
> 17/04/20 21:10:45 INFO scheduler.DAGScheduler: ResultStage 0 (collect at BoundedDataset.java:88) failed in 1.260 s
> 17/04/20 21:10:45 INFO scheduler.DAGScheduler: Job 0 failed: collect at BoundedDataset.java:88, took 1.424075 s
> 17/04/20 21:10:46 ERROR yarn.ApplicationMaster: User class threw exception: org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.InvalidClassException: org.apache.beam.runners.spark.coders.CoderHelpers$3; local class incompatible: stream classdesc serialVersionUID = 1334222146820528045, local class serialVersionUID = 5119956493581628999
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.InvalidClassException: org.apache.beam.runners.spark.coders.CoderHelpers$3; local class incompatible: stream classdesc serialVersionUID = 1334222146820528045, local class serialVersionUID = 5119956493581628999
> at org.apache.beam.runners.spark.SparkPipelineResult.beamExceptionFrom(SparkPipelineResult.java:73)
> at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:113)
> at org.apache.beam.runners.spark.SparkPipelineResult.waitUntilFinish(SparkPipelineResult.java:102)
> at com.chinamobile.cmss.etl.beam.transform.server.SparkMain.main(SparkMain.java:96)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:558)
> Caused by: java.io.InvalidClassException: org.apache.beam.runners.spark.coders.CoderHelpers$3; local class incompatible: stream classdesc serialVersionUID = 1334222146820528045, local class serialVersionUID = 5119956493581628999
> at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)
> at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)
> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)
> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)