You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Mark Liu (JIRA)" <ji...@apache.org> on 2018/11/07 19:48:00 UTC
[jira] [Comment Edited] (BEAM-5953) Support DataflowRunner on
Python 3
[ https://issues.apache.org/jira/browse/BEAM-5953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16677546#comment-16677546 ]
Mark Liu edited comment on BEAM-5953 at 11/7/18 7:47 PM:
---------------------------------------------------------
With provided Python 3 SDK container (by BEAM-5089) and some fix ([https://github.com/markflyhigh/incubator-beam/pull/3]) to the Python 3 type error, I'm able to invoke wordcount_fnapi_it against TestDataflowRunner on Python 3. The job can be submitted to the service but the runner harness seems broken.
Failure job link:
[https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-05_15_43_59-9596490965399700763?project=google.com:clouddfe]
Exception in worker log:
{code:java}
I Exception in thread "main"
I org.apache.beam.vendor.protobuf.v3.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:115)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:551)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.GeneratedMessageV3.parseUnknownFieldProto3(GeneratedMessageV3.java:305)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform.<init>(RunnerApi.java:7084)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform.<init>(RunnerApi.java:6978)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform$1.parsePartialFrom(RunnerApi.java:9169)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform$1.parsePartialFrom(RunnerApi.java:9163)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform$Builder.mergeFrom(RunnerApi.java:8052)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform$Builder.mergeFrom(RunnerApi.java:7835)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2408)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntryLite.parseField(MapEntryLite.java:128)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntryLite.parseEntry(MapEntryLite.java:184)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntry.<init>(MapEntry.java:106)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntry.<init>(MapEntry.java:50)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntry$Metadata$1.parsePartialFrom(MapEntry.java:70)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntry$Metadata$1.parsePartialFrom(MapEntry.java:64)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2424)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Components.<init>(RunnerApi.java:343)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Components.<init>(RunnerApi.java:300)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Components$1.parsePartialFrom(RunnerApi.java:2166)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Components$1.parsePartialFrom(RunnerApi.java:2160)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2424)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline.<init>(RunnerApi.java:5523)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline.<init>(RunnerApi.java:5481)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline$1.parsePartialFrom(RunnerApi.java:6612)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline$1.parsePartialFrom(RunnerApi.java:6606)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:221)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:239)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:244)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.GeneratedMessageV3.parseWithIOException(GeneratedMessageV3.java:311)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline.parseFrom(RunnerApi.java:5853)
I at org.apache.beam.runners.dataflow.worker.DataflowWorkerHarnessHelper.getPipelineFromEnv(DataflowWorkerHarnessHelper.java:117)
I at org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:58)
I java failed with exit status 1
F Harness failed: exit status 1
{code}
was (Author: markflyhigh):
With provided Python 3 SDK container (by BEAM-5089) and some fix (https://github.com/markflyhigh/incubator-beam/pull/3) to the type error, I'm able to invoke wordcount_fnapi_it against TestDataflowRunner on Python 3. The job is submitted to the service successfully but worker didn't start properly.
Failure job link:
https://pantheon.corp.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-05_15_43_59-9596490965399700763?project=google.com:clouddfe
Exception in worker log:
{code}
I Exception in thread "main"
I org.apache.beam.vendor.protobuf.v3.com.google.protobuf.InvalidProtocolBufferException$InvalidWireTypeException: Protocol message tag had invalid wire type.
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.InvalidProtocolBufferException.invalidWireType(InvalidProtocolBufferException.java:115)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.UnknownFieldSet$Builder.mergeFieldFrom(UnknownFieldSet.java:551)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.GeneratedMessageV3.parseUnknownFieldProto3(GeneratedMessageV3.java:305)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform.<init>(RunnerApi.java:7084)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform.<init>(RunnerApi.java:6978)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform$1.parsePartialFrom(RunnerApi.java:9169)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform$1.parsePartialFrom(RunnerApi.java:9163)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform$Builder.mergeFrom(RunnerApi.java:8052)
I at org.apache.beam.model.pipeline.v1.RunnerApi$PTransform$Builder.mergeFrom(RunnerApi.java:7835)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2408)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntryLite.parseField(MapEntryLite.java:128)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntryLite.parseEntry(MapEntryLite.java:184)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntry.<init>(MapEntry.java:106)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntry.<init>(MapEntry.java:50)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntry$Metadata$1.parsePartialFrom(MapEntry.java:70)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.MapEntry$Metadata$1.parsePartialFrom(MapEntry.java:64)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2424)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Components.<init>(RunnerApi.java:343)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Components.<init>(RunnerApi.java:300)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Components$1.parsePartialFrom(RunnerApi.java:2166)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Components$1.parsePartialFrom(RunnerApi.java:2160)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2424)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline.<init>(RunnerApi.java:5523)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline.<init>(RunnerApi.java:5481)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline$1.parsePartialFrom(RunnerApi.java:6612)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline$1.parsePartialFrom(RunnerApi.java:6606)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:221)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:239)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:244)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.AbstractParser.parseFrom(AbstractParser.java:49)
I at org.apache.beam.vendor.protobuf.v3.com.google.protobuf.GeneratedMessageV3.parseWithIOException(GeneratedMessageV3.java:311)
I at org.apache.beam.model.pipeline.v1.RunnerApi$Pipeline.parseFrom(RunnerApi.java:5853)
I at org.apache.beam.runners.dataflow.worker.DataflowWorkerHarnessHelper.getPipelineFromEnv(DataflowWorkerHarnessHelper.java:117)
I at org.apache.beam.runners.dataflow.worker.DataflowRunnerHarness.main(DataflowRunnerHarness.java:58)
I java failed with exit status 1
F Harness failed: exit status 1
{code}
> Support DataflowRunner on Python 3
> ----------------------------------
>
> Key: BEAM-5953
> URL: https://issues.apache.org/jira/browse/BEAM-5953
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Mark Liu
> Assignee: Mark Liu
> Priority: Major
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)