You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Tom Barber <to...@spicule.co.uk> on 2019/09/17 00:02:36 UTC

Running against Spark 2.2.3

Hello folks,

Spark question, both Python and Java gets sad against the portable runner
when trying to run against an existing Spark 2.2.3 server, which the docs
say is supported.

The Spark logs say:

19/09/16 23:59:48 ERROR TransportRequestHandler: Error while invoking
RpcHandler#receive() for one-way message.
java.io.InvalidClassException: org.apache.spark.rpc.RpcEndpointRef; local
class incompatible: stream classdesc serialVersionUID =
-1329125091869941550, local class serialVersionUID = 1835832137613908542

Which from previous experience is generally an library/scala
incompatibility, but I’m not sure what, if anything, I can do to the
portable runner to make it happy.

Thanks

Tom

-- 


Spicule Limited is registered in England & Wales. Company Number: 
09954122. Registered office: First Floor, Telecom House, 125-135 Preston 
Road, Brighton, England, BN1 6AF. VAT No. 251478891.




All engagements 
are subject to Spicule Terms and Conditions of Business. This email and its 
contents are intended solely for the individual to whom it is addressed and 
may contain information that is confidential, privileged or otherwise 
protected from disclosure, distributing or copying. Any views or opinions 
presented in this email are solely those of the author and do not 
necessarily represent those of Spicule Limited. The company accepts no 
liability for any damage caused by any virus transmitted by this email. If 
you have received this message in error, please notify us immediately by 
reply email before deleting it from your system. Service of legal notice 
cannot be effected on Spicule Limited by email.

Re: Running against Spark 2.2.3

Posted by Tom Barber <to...@spicule.co.uk>.
I downgraded the Spark version in the 2.16 branch and now the job starts,
but it seems whatever I do I get:

Caused by:
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException:
UNIMPLEMENTED: Method not found!
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:235)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:216)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:141)
at
org.apache.beam.model.fnexecution.v1.BeamFnExternalWorkerPoolGrpc$BeamFnExternalWorkerPoolBlockingStub.startWorker(BeamFnExternalWorkerPoolGrpc.java:226)

Sad times!


On 17 September 2019 at 01:02:36, Tom Barber (tom@spicule.co.uk) wrote:

Hello folks,

Spark question, both Python and Java gets sad against the portable runner
when trying to run against an existing Spark 2.2.3 server, which the docs
say is supported.

The Spark logs say:

19/09/16 23:59:48 ERROR TransportRequestHandler: Error while invoking
RpcHandler#receive() for one-way message.
java.io.InvalidClassException: org.apache.spark.rpc.RpcEndpointRef; local
class incompatible: stream classdesc serialVersionUID =
-1329125091869941550, local class serialVersionUID = 1835832137613908542

Which from previous experience is generally an library/scala
incompatibility, but I’m not sure what, if anything, I can do to the
portable runner to make it happy.

Thanks

Tom

-- 


Spicule Limited is registered in England & Wales. Company Number: 
09954122. Registered office: First Floor, Telecom House, 125-135 Preston 
Road, Brighton, England, BN1 6AF. VAT No. 251478891.




All engagements 
are subject to Spicule Terms and Conditions of Business. This email and its 
contents are intended solely for the individual to whom it is addressed and 
may contain information that is confidential, privileged or otherwise 
protected from disclosure, distributing or copying. Any views or opinions 
presented in this email are solely those of the author and do not 
necessarily represent those of Spicule Limited. The company accepts no 
liability for any damage caused by any virus transmitted by this email. If 
you have received this message in error, please notify us immediately by 
reply email before deleting it from your system. Service of legal notice 
cannot be effected on Spicule Limited by email.