Java object serialization error, java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible

[Elliot Metsger <em...@gmail.com>]

Howdy folks, super-new to Beam, and attempting to get a simple example
working with Go, using the portable runner and Spark. There seems to be an
incompatibility between Java components, and I’m not quite sure where the
disconnect is, but at the root it seems to be an incompatibility with
object serializations.

When I submit the job via the go sdk, it errors out on the Spark side with:
[8:59 AM] 22/08/25 12:45:59 ERROR TransportRequestHandler: Error while
invoking RpcHandler#receive() for one-way message.
java.io.InvalidClassException:
org.apache.spark.deploy.ApplicationDescription; local class incompatible:
stream classdesc serialVersionUID = 6543101073799644159, local class
serialVersionUID = 1574364215946805297
I’m using apache/beam_spark_job_server:2.41.0 and apache/spark:latest.
 (docker-compose[0], hello world wordcount example pipeline[1]).

Any ideas on where to look?  It looks like the Beam JobService is using
Java 8 (?) and Spark is using Java 11.  I’ve tried downgrading Spark from
3.3.0 to 3.1.3 (the earliest version for which Docker images are
available), and downgrading to Beam 2.40.0 with no luck.

This simple repo[2] should demonstrate the issue.  Any pointers would be
appreciated!

[0]: https://github.com/emetsger/beam-test/blob/develop/docker-compose.yml
[1]:
https://github.com/emetsger/beam-test/blob/develop/debugging_wordcount.go
[2]: https://github.com/emetsger/beam-test