[jira] [Created] (BEAM-3204) Coders only should have a FunctionSpec, not an SdkFunctionSpec

["Kenneth Knowles (JIRA)" <ji...@apache.org>]

Kenneth Knowles created BEAM-3204:
-------------------------------------

             Summary: Coders only should have a FunctionSpec, not an SdkFunctionSpec
                 Key: BEAM-3204
                 URL: https://issues.apache.org/jira/browse/BEAM-3204
             Project: Beam
          Issue Type: Improvement
          Components: beam-model
            Reporter: Kenneth Knowles
            Assignee: Henning Rohde


We added environments to coders to account for "custom" coders where it is only really possible for one SDK to understand them, like this:

{code}
Coder {
  spec: SdkFunctionSpec {
    environment: "java_sdk_docker_container",
    spec: FunctionSpec {
      urn: "beam:coder:java_custom_coder",
      payload: <serialized java bytes>
    }
  }
}
{code}

But a coder must be understood by both the producer of a PCollection and its consumers. A coder is not the same as other UDF, though these are user-defined.

A pipeline where either the producer or consumer cannot handle the coder is invalid, and we will have to build our cross-language APIs to prevent construction of such a pipeline. So we can drop the environment.

I think there are some folks who want to reserve the ability to add an environment later, perhaps, to not pain ourselves into a corner. In this case, we can just add a field to Coder.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)