You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2021/10/16 17:25:00 UTC
[jira] [Commented] (BEAM-12762) java.io.InvalidClassException with
Spark 3.1.2
[ https://issues.apache.org/jira/browse/BEAM-12762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17429594#comment-17429594 ]
Beam JIRA Bot commented on BEAM-12762:
--------------------------------------
This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3.
Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean.
> java.io.InvalidClassException with Spark 3.1.2
> ----------------------------------------------
>
> Key: BEAM-12762
> URL: https://issues.apache.org/jira/browse/BEAM-12762
> Project: Beam
> Issue Type: Bug
> Components: runner-spark
> Reporter: Kyle Weaver
> Priority: P2
> Labels: stale-P2
>
> This was reported on the mailing list.
>
> ----
>
> Using spark downloaded from below link,
>
> [https://www.apache.org/dyn/closer.lua/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz]
>
> I get below error when submitting a pipeline.
> Full error is on [https://gist.github.com/yuwtennis/7b0c1dc0dcf98297af1e3179852ca693].
>
> ------------------------------------------------------------------------------------------------------------------
> 21/08/16 01:10:26 WARN TransportChannelHandler: Exception in connection from /[192.168.11.2:35601|http://192.168.11.2:35601/]
> java.io.InvalidClassException: scala.collection.mutable.WrappedArray$ofRef; local class incompatible: stream classdesc serialVersionUID = 3456489343829468865, local class serialVersionUID = 1028182004549731694
> at java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689)
> ...
> ------------------------------------------------------------------------------------------------------------------
>
> SDK Harness and Job service are deployed as below.
>
> 1. SDK Harness
>
> sudo docker run --net=host apache/beam_spark3_job_server:2.31.0 --spark-master-url=spark://localhost:7077 --clean-artifacts-per-job true
>
> 2. Job service
>
> sudo docker run --net=host apache/beam_python3.8_sdk:2.31.0 --worker_pool
>
> * apache/beam_spark_job_server:2.31.0 for spark 2.4.8
>
> 3. SDK client code
>
> [https://gist.github.com/yuwtennis/2e4c13c79f71e8f713e947955115b3e2]
> Spark 2.4.8 succeeded without any errors using above components.
>
> [https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)