You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kenneth Knowles (JIRA)" <ji...@apache.org> on 2019/03/06 17:16:00 UTC
[jira] [Updated] (BEAM-4582) Incorrectly translates
apache_beam.runners.dataflow.native_io.streaming_create.DecodeAndEmitDoFn
when creating the Dataflow pipeline json description
[ https://issues.apache.org/jira/browse/BEAM-4582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kenneth Knowles updated BEAM-4582:
----------------------------------
Labels: portability triaged (was: portability)
> Incorrectly translates apache_beam.runners.dataflow.native_io.streaming_create.DecodeAndEmitDoFn when creating the Dataflow pipeline json description
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: BEAM-4582
> URL: https://issues.apache.org/jira/browse/BEAM-4582
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Reporter: Luke Cwik
> Priority: Major
> Labels: portability, triaged
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> When executing against Dataflow, the JSON pipeline description contains the following JSON which doesn't appear in the pipeline proto:
>
> {code:java}
> {
> "kind": "ParallelDo",
> "name": "s2",
> "properties": {
> "display_data": [
> {
> "key": "fn",
> "label": "Transform Function",
> "namespace": "apache_beam.transforms.core.ParDo",
> "shortValue": "DecodeAndEmitDoFn",
> "type": "STRING",
> "value": "apache_beam.runners.dataflow.native_io.streaming_create.DecodeAndEmitDoFn"
> }
> ],
> "non_parallel_inputs": {},
> "output_info": [
> {
> "encoding": {
> "@type": "kind:windowed_value",
> "component_encodings": [
> {
> "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
> "component_encodings": [
> {
> "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
> "component_encodings": []
> },
> {
> "@type": "FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
> "component_encodings": []
> }
> ],
> "is_pair_like": true
> },
> {
> "@type": "kind:global_window"
> }
> ],
> "is_wrapper": true
> },
> "output_name": "out",
> "user_name": "Some Numbers/Decode Values.out"
> }
> ],
> "parallel_input": {
> "@type": "OutputReference",
> "output_name": "out",
> "step_name": "s1"
> },
> "serialized_fn": "ref_AppliedPTransform_AppliedPTransform_45",
> "user_name": "Some Numbers/Decode Values"
> }
> },
> {code}
> This causes the DataflowRunner to use a legacy code path and ask the Python SDK harness to execute a transform with a payload *ref_AppliedPTransform_AppliedPTransform_45* instead of sending the PTransform proto.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)