[jira] [Created] (BEAM-4689) Dataflow postcommit broken

["Kenneth Knowles (JIRA)" <ji...@apache.org>]

Kenneth Knowles created BEAM-4689:
-------------------------------------

             Summary: Dataflow postcommit broken
                 Key: BEAM-4689
                 URL: https://issues.apache.org/jira/browse/BEAM-4689
             Project: Beam
          Issue Type: New Feature
          Components: runner-dataflow
            Reporter: Kenneth Knowles
            Assignee: Thomas Groh


The Dataflow postcommit is broken in a way that seems real and user-impacting:

https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/testReport/junit/org.apache.beam.sdk.transforms/SplittableDoFnTest/testSideInput/

{code}
Caused by: java.lang.IllegalArgumentException: unable to deserialize Serialized DoFnInfo
        ...
Caused by: java.io.InvalidClassException: org.apache.beam.runners.core.construction.SplittableParDo$RandomUniqueKeyFn; local class incompatible: stream classdesc serialVersionUID = 6068396661487412884, local class serialVersionUID = -617521663543732196
{code}

This means that the worker is using a version of the class from its own classpath, not the version from the user's staged pipeline. It implies that the worker is not shading runners-core-construction. Because that is where a ton of utility DoFns live, it is critical that it be shaded.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)