You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Beam JIRA Bot (Jira)" <ji...@apache.org> on 2021/05/30 17:20:01 UTC
[jira] [Commented] (BEAM-12256) PubsubIO.readAvroGenericRecord
creates SchemaCoder that fails to decode some Avro logical types
[ https://issues.apache.org/jira/browse/BEAM-12256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17354079#comment-17354079 ]
Beam JIRA Bot commented on BEAM-12256:
--------------------------------------
This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned.
> PubsubIO.readAvroGenericRecord creates SchemaCoder that fails to decode some Avro logical types
> -----------------------------------------------------------------------------------------------
>
> Key: BEAM-12256
> URL: https://issues.apache.org/jira/browse/BEAM-12256
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp, sdk-java-core
> Affects Versions: 2.29.0
> Reporter: Brian Hulette
> Assignee: Brian Hulette
> Priority: P1
> Labels: stale-assigned
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For example, when PubsubIO.readAvroGenericRecord is used with an avro schema that includes a decimal type with a non-zero scale, it can encounter the following exception when decoding:
> {code:java}
> org.apache.avro.AvroTypeException: Cannot encode decimal with scale 17 as scale 0
> at org.apache.avro.Conversions$DecimalConversion.toBytes(Conversions.java:92)
> at org.apache.beam.sdk.schemas.utils.AvroUtils.genericFromBeamField(AvroUtils.java:975)
> at org.apache.beam.sdk.schemas.utils.AvroUtils.toGenericRecord(AvroUtils.java:397)
> at org.apache.beam.sdk.schemas.utils.AvroUtils$RowToGenericRecordFn.apply(AvroUtils.java:547)
> at org.apache.beam.sdk.schemas.utils.AvroUtils$RowToGenericRecordFn.apply(AvroUtils.java:538)
> at org.apache.beam.sdk.schemas.SchemaCoder.decode(SchemaCoder.java:123)
> at org.apache.beam.sdk.io.gcp.bigquery.TableRowInfoCoder.decode(TableRowInfoCoder.java:64)
> at org.apache.beam.sdk.io.gcp.bigquery.TableRowInfoCoder.decode(TableRowInfoCoder.java:30)
> at org.apache.beam.runners.dataflow.worker.WindmillKeyedWorkItem.lambda$elementsIterable$2(WindmillKeyedWorkItem.java:112)
> at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Iterators$6.transform(Iterators.java:785)
> at org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.TransformedIterator.next(TransformedIterator.java:47)
> at org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowReshuffleFn.processElement(StreamingGroupAlsoByWindowReshuffleFn.java:56)
> at org.apache.beam.runners.dataflow.worker.StreamingGroupAlsoByWindowReshuffleFn.processElement(StreamingGroupAlsoByWindowReshuffleFn.java:39)
> at org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.invokeProcessElement(GroupAlsoByWindowFnRunner.java:121)
> at org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowFnRunner.processElement(GroupAlsoByWindowFnRunner.java:73)
> at org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowsParDoFn.processElement(GroupAlsoByWindowsParDoFn.java:137)
> at org.apache.beam.runners.dataflow.worker.util.common.worker.ParDoOperation.process(ParDoOperation.java:44)
> at org.apache.beam.runners.dataflow.worker.util.common.worker.OutputReceiver.process(OutputReceiver.java:49)
> at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:212)
> at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start(ReadOperation.java:163)
> at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:92)
> at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process(StreamingDataflowWorker.java:1426)
> at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1100(StreamingDataflowWorker.java:163)
> at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$7.run(StreamingDataflowWorker.java:1105)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)